2024-03-25T16:03:03.003 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:03:03.003 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621241 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621241/orig.config.yaml 2024-03-25T16:03:03.048 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:03:03.129 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621241 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621241' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi018.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM5iihOi7mgKna/7JMuqrPgo58gKUIVlTPVjbWJCGi0Pig7Q+zNrm0Y3aQEAt+gWMd16I3LFzvsidKHn5+fpqlU= smithi169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGqazL4b6MWoRUTI63i1G8u0ySRHWPcHkfVSpRawEMRw0lsdpluWX2Sf71VruWn3bWzc89YLVmC5mUHvNm88rcQ= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:03:03.129 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:03:03.130 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:03:03.130 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:03:03.132 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:03:03.157 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:03:03.157 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:03:03.157 INFO:teuthology.packaging:ref: None 2024-03-25T16:03:03.157 INFO:teuthology.packaging:tag: None 2024-03-25T16:03:03.157 INFO:teuthology.packaging:branch: main 2024-03-25T16:03:03.157 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:03:03.157 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:03:03.509 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:03:03.512 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:03:03.520 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:03:03.520 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:03:03.530 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:03:03.551 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:03:03.568 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:03:03.603 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi018.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621241', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 15:57:59.039127', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM5iihOi7mgKna/7JMuqrPgo58gKUIVlTPVjbWJCGi0Pig7Q+zNrm0Y3aQEAt+gWMd16I3LFzvsidKHn5+fpqlU='} 2024-03-25T16:03:03.628 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621241', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 15:57:59.041045', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGqazL4b6MWoRUTI63i1G8u0ySRHWPcHkfVSpRawEMRw0lsdpluWX2Sf71VruWn3bWzc89YLVmC5mUHvNm88rcQ='} 2024-03-25T16:03:03.628 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:03:03.637 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:03:03.638 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:03:03.638 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:03:03.740 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9897cd80d0>, signals=[15]) 2024-03-25T16:03:03.741 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:03:03.750 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:03:03.750 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T16:03:03.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:03:03.863 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T16:03:03.864 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:03:03.962 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:03:03.969 DEBUG:teuthology.orchestra.run.smithi018:> uname -m 2024-03-25T16:03:03.986 INFO:teuthology.orchestra.run.smithi018.stdout:x86_64 2024-03-25T16:03:03.986 DEBUG:teuthology.orchestra.run.smithi018:> cat /etc/os-release 2024-03-25T16:03:04.044 INFO:teuthology.orchestra.run.smithi018.stdout:NAME="CentOS Stream" 2024-03-25T16:03:04.044 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION="9" 2024-03-25T16:03:04.044 INFO:teuthology.orchestra.run.smithi018.stdout:ID="centos" 2024-03-25T16:03:04.044 INFO:teuthology.orchestra.run.smithi018.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:03:04.044 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_ID="9" 2024-03-25T16:03:04.045 INFO:teuthology.orchestra.run.smithi018.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:03:04.045 INFO:teuthology.orchestra.run.smithi018.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:03:04.045 INFO:teuthology.orchestra.run.smithi018.stdout:ANSI_COLOR="0;31" 2024-03-25T16:03:04.045 INFO:teuthology.orchestra.run.smithi018.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:03:04.045 INFO:teuthology.orchestra.run.smithi018.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:03:04.045 INFO:teuthology.orchestra.run.smithi018.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:03:04.045 INFO:teuthology.orchestra.run.smithi018.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:03:04.045 INFO:teuthology.orchestra.run.smithi018.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:03:04.045 INFO:teuthology.orchestra.run.smithi018.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:03:04.046 INFO:teuthology.lock.ops:Updating smithi018.front.sepia.ceph.com on lock server 2024-03-25T16:03:04.070 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2024-03-25T16:03:04.088 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2024-03-25T16:03:04.088 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2024-03-25T16:03:04.144 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Stream" 2024-03-25T16:03:04.144 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="9" 2024-03-25T16:03:04.145 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos" 2024-03-25T16:03:04.145 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:03:04.145 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="9" 2024-03-25T16:03:04.145 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:03:04.145 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:03:04.145 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2024-03-25T16:03:04.145 INFO:teuthology.orchestra.run.smithi169.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:03:04.145 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:03:04.145 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:03:04.146 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:03:04.146 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:03:04.146 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:03:04.146 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2024-03-25T16:03:04.170 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:03:04.190 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:03:04.199 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:03:04.200 DEBUG:teuthology.orchestra.run.smithi018:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:03:04.203 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:03:04.220 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:03:04.230 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:03:04.230 DEBUG:teuthology.orchestra.run.smithi018:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:03:04.263 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:03:04.388 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:03:04.451 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:03:04.468 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:03:04.469 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:03:04.469 DEBUG:teuthology.orchestra.run.smithi018:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:03:04.471 DEBUG:teuthology.orchestra.run.smithi169:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:03:04.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:03:04.488 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-03-25T16:03:04.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:03:04.490 DEBUG:teuthology.orchestra.run.smithi018:> uname -r 2024-03-25T16:03:04.543 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:03:04.543 INFO:teuthology.task.kernel:Running kernel on smithi169: 5.14.0-361.el9.x86_64 2024-03-25T16:03:04.543 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum install -y kernel 2024-03-25T16:03:04.549 INFO:teuthology.orchestra.run.smithi018.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:03:04.549 INFO:teuthology.task.kernel:Running kernel on smithi018: 5.14.0-361.el9.x86_64 2024-03-25T16:03:04.549 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum install -y kernel 2024-03-25T16:03:07.022 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - AppStream 9.3 MB/s | 6.7 MB 00:00 2024-03-25T16:03:07.026 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS-9 - AppStream 8.4 MB/s | 6.7 MB 00:00 2024-03-25T16:03:08.398 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-03-25T16:03:08.500 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-25T16:03:09.271 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-25T16:03:10.252 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.0 MB 00:01 2024-03-25T16:03:11.148 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-25T16:03:11.901 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - AppStream 33 MB/s | 19 MB 00:00 2024-03-25T16:03:15.040 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-25T16:03:16.532 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-25T16:03:17.026 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.1 MB 00:01 2024-03-25T16:03:18.111 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-25T16:03:18.518 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-03-25T16:03:22.870 INFO:teuthology.orchestra.run.smithi018.stdout:Extra Packages for Enterprise Linux 4.9 MB/s | 21 MB 00:04 2024-03-25T16:03:23.067 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-25T16:03:25.174 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:03:25.174 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:03:25.267 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:03:25.278 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:03:25.278 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-03-25T16:03:25.278 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:03:25.279 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:03:25.279 INFO:teuthology.orchestra.run.smithi169.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:03:25.279 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T16:03:25.279 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:03:25.279 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:03:25.279 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:03:25.279 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:03:25.279 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:03:25.279 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:03:25.279 INFO:teuthology.orchestra.run.smithi169.stdout:Install 4 Packages 2024-03-25T16:03:25.279 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:03:25.280 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 76 M 2024-03-25T16:03:25.280 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 123 M 2024-03-25T16:03:25.280 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:03:25.976 INFO:teuthology.orchestra.run.smithi169.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 70 kB/s | 44 kB 00:00 2024-03-25T16:03:28.053 INFO:teuthology.orchestra.run.smithi018.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-25T16:03:30.244 INFO:teuthology.orchestra.run.smithi018.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:03:30.244 INFO:teuthology.orchestra.run.smithi018.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:03:30.336 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:03:30.348 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:03:30.348 INFO:teuthology.orchestra.run.smithi018.stdout: Package Architecture Version Repository Size 2024-03-25T16:03:30.348 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:03:30.348 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:03:30.349 INFO:teuthology.orchestra.run.smithi018.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:03:30.349 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T16:03:30.349 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:03:30.349 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:03:30.349 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:03:30.349 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:03:30.349 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:03:30.349 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:03:30.350 INFO:teuthology.orchestra.run.smithi018.stdout:Install 4 Packages 2024-03-25T16:03:30.350 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:03:30.350 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 76 M 2024-03-25T16:03:30.350 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 123 M 2024-03-25T16:03:30.350 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:03:31.129 INFO:teuthology.orchestra.run.smithi018.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 61 kB/s | 44 kB 00:00 2024-03-25T16:03:46.216 INFO:teuthology.orchestra.run.smithi169.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 731 kB/s | 15 MB 00:20 2024-03-25T16:03:50.477 INFO:teuthology.orchestra.run.smithi018.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 760 kB/s | 15 MB 00:20 2024-03-25T16:04:00.889 INFO:teuthology.orchestra.run.smithi169.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 806 kB/s | 27 MB 00:34 2024-03-25T16:04:01.633 INFO:teuthology.orchestra.run.smithi018.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 922 kB/s | 27 MB 00:30 2024-03-25T16:04:02.059 INFO:teuthology.orchestra.run.smithi018.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 1.1 MB/s | 33 MB 00:31 2024-03-25T16:04:02.059 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:04:02.059 INFO:teuthology.orchestra.run.smithi018.stdout:Total 2.4 MB/s | 76 MB 00:31 2024-03-25T16:04:02.750 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:04:03.176 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:04:03.176 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:04:04.217 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:04:04.217 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:04:06.822 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:04:07.439 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:04:07.526 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:04:09.747 INFO:teuthology.orchestra.run.smithi169.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 771 kB/s | 33 MB 00:44 2024-03-25T16:04:09.902 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:04:09.903 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.7 MB/s | 76 MB 00:44 2024-03-25T16:04:10.383 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:04:10.410 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:04:10.521 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:04:10.811 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:04:10.811 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:04:11.875 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:04:11.875 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:04:14.334 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:04:15.054 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:04:15.139 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:04:15.596 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:04:15.638 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:17.924 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:04:18.068 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:04:20.728 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:22.810 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:04:22.843 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:27.593 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:51.366 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:51.396 INFO:teuthology.orchestra.run.smithi018.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:04:51.396 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:04:51.411 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:51.772 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:51.772 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:04:51.772 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:04:51.772 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:04:52.207 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:52.207 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:04:52.207 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:04:52.207 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:52.207 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:04:52.207 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:04:52.207 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:04:52.207 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:04:52.207 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:04:52.539 DEBUG:teuthology.orchestra.run.smithi018:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:04:52.688 INFO:teuthology.orchestra.run.smithi018.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:52.688 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:04:52.688 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:04:52.688 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:04:52.688 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:04:52.689 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:04:52.689 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi018.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:04:52.689 DEBUG:teuthology.orchestra.run.smithi018:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:04:52.771 INFO:teuthology.orchestra.run.smithi018.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:52.771 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:04:52.772 DEBUG:teuthology.orchestra.run.smithi018:> sudo rpm -qi grub2-tools 2024-03-25T16:04:52.809 INFO:teuthology.orchestra.run.smithi018.stdout:Name : grub2-tools 2024-03-25T16:04:52.809 INFO:teuthology.orchestra.run.smithi018.stdout:Epoch : 1 2024-03-25T16:04:52.809 INFO:teuthology.orchestra.run.smithi018.stdout:Version : 2.06 2024-03-25T16:04:52.809 INFO:teuthology.orchestra.run.smithi018.stdout:Release : 68.el9 2024-03-25T16:04:52.810 INFO:teuthology.orchestra.run.smithi018.stdout:Architecture: x86_64 2024-03-25T16:04:52.810 INFO:teuthology.orchestra.run.smithi018.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:04:52.810 INFO:teuthology.orchestra.run.smithi018.stdout:Group : Unspecified 2024-03-25T16:04:52.810 INFO:teuthology.orchestra.run.smithi018.stdout:Size : 8264254 2024-03-25T16:04:52.810 INFO:teuthology.orchestra.run.smithi018.stdout:License : GPLv3+ 2024-03-25T16:04:52.810 INFO:teuthology.orchestra.run.smithi018.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:04:52.810 INFO:teuthology.orchestra.run.smithi018.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:04:52.810 INFO:teuthology.orchestra.run.smithi018.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:04:52.810 INFO:teuthology.orchestra.run.smithi018.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:04:52.810 INFO:teuthology.orchestra.run.smithi018.stdout:Packager : builder@centos.org 2024-03-25T16:04:52.810 INFO:teuthology.orchestra.run.smithi018.stdout:Vendor : CentOS 2024-03-25T16:04:52.810 INFO:teuthology.orchestra.run.smithi018.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:04:52.810 INFO:teuthology.orchestra.run.smithi018.stdout:Summary : Support tools for GRUB. 2024-03-25T16:04:52.810 INFO:teuthology.orchestra.run.smithi018.stdout:Description : 2024-03-25T16:04:52.811 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:04:52.811 INFO:teuthology.orchestra.run.smithi018.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:04:52.811 INFO:teuthology.orchestra.run.smithi018.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:04:52.811 INFO:teuthology.orchestra.run.smithi018.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:04:52.811 INFO:teuthology.orchestra.run.smithi018.stdout:hardware devices. 2024-03-25T16:04:52.811 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:04:52.811 INFO:teuthology.orchestra.run.smithi018.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:04:52.812 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:04:52.812 INFO:teuthology.task.kernel:Updating grub on smithi018 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:04:52.812 DEBUG:teuthology.orchestra.run.smithi018:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:04:53.432 INFO:teuthology.orchestra.run.smithi018.stderr:Generating grub configuration file ... 2024-03-25T16:04:54.830 INFO:teuthology.orchestra.run.smithi018.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:04:54.877 INFO:teuthology.orchestra.run.smithi018.stderr:done 2024-03-25T16:04:54.879 DEBUG:teuthology.orchestra.run.smithi018:> mktemp 2024-03-25T16:04:54.895 INFO:teuthology.orchestra.run.smithi018.stdout:/tmp/tmp.Wk96yZYZeS 2024-03-25T16:04:54.895 DEBUG:teuthology.orchestra.run.smithi018:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Wk96yZYZeS 2024-03-25T16:04:54.974 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0666 /tmp/tmp.Wk96yZYZeS 2024-03-25T16:04:55.112 DEBUG:teuthology.orchestra.remote:smithi018:/tmp/tmp.Wk96yZYZeS is 6KB 2024-03-25T16:04:55.161 DEBUG:teuthology.orchestra.run.smithi018:> rm -fr /tmp/tmp.Wk96yZYZeS 2024-03-25T16:04:55.176 DEBUG:teuthology.orchestra.run.smithi018:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:04:55.245 INFO:teuthology.orchestra.run.smithi018.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:04:55.245 INFO:teuthology.orchestra.run.smithi018.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:04:55.246 INFO:teuthology.orchestra.run.smithi018.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:04:55.246 INFO:teuthology.orchestra.run.smithi018.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:04:55.247 DEBUG:teuthology.orchestra.run.smithi018:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:04:55.402 DEBUG:teuthology.orchestra.run.smithi018:> sudo shutdown -r now 2024-03-25T16:04:55.496 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-03-25T16:04:57.435 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:57.435 INFO:teuthology.orchestra.run.smithi169.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:04:57.436 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:04:57.479 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:57.903 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:04:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:04:57.904 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:04:58.319 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:58.319 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:04:58.319 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:04:58.319 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:58.319 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:04:58.319 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:04:58.319 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:04:58.319 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:04:58.319 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:04:58.647 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:04:58.752 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:58.752 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:04:58.753 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:04:58.753 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:04:58.753 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:04:58.753 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:04:58.753 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi169.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:04:58.753 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:04:58.834 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:58.834 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:04:58.834 DEBUG:teuthology.orchestra.run.smithi169:> sudo rpm -qi grub2-tools 2024-03-25T16:04:58.874 INFO:teuthology.orchestra.run.smithi169.stdout:Name : grub2-tools 2024-03-25T16:04:58.875 INFO:teuthology.orchestra.run.smithi169.stdout:Epoch : 1 2024-03-25T16:04:58.875 INFO:teuthology.orchestra.run.smithi169.stdout:Version : 2.06 2024-03-25T16:04:58.875 INFO:teuthology.orchestra.run.smithi169.stdout:Release : 68.el9 2024-03-25T16:04:58.875 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: x86_64 2024-03-25T16:04:58.875 INFO:teuthology.orchestra.run.smithi169.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:04:58.875 INFO:teuthology.orchestra.run.smithi169.stdout:Group : Unspecified 2024-03-25T16:04:58.875 INFO:teuthology.orchestra.run.smithi169.stdout:Size : 8264254 2024-03-25T16:04:58.875 INFO:teuthology.orchestra.run.smithi169.stdout:License : GPLv3+ 2024-03-25T16:04:58.875 INFO:teuthology.orchestra.run.smithi169.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:04:58.875 INFO:teuthology.orchestra.run.smithi169.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:04:58.876 INFO:teuthology.orchestra.run.smithi169.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:04:58.876 INFO:teuthology.orchestra.run.smithi169.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:04:58.876 INFO:teuthology.orchestra.run.smithi169.stdout:Packager : builder@centos.org 2024-03-25T16:04:58.876 INFO:teuthology.orchestra.run.smithi169.stdout:Vendor : CentOS 2024-03-25T16:04:58.876 INFO:teuthology.orchestra.run.smithi169.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:04:58.876 INFO:teuthology.orchestra.run.smithi169.stdout:Summary : Support tools for GRUB. 2024-03-25T16:04:58.876 INFO:teuthology.orchestra.run.smithi169.stdout:Description : 2024-03-25T16:04:58.876 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:04:58.876 INFO:teuthology.orchestra.run.smithi169.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:04:58.876 INFO:teuthology.orchestra.run.smithi169.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:04:58.876 INFO:teuthology.orchestra.run.smithi169.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:04:58.877 INFO:teuthology.orchestra.run.smithi169.stdout:hardware devices. 2024-03-25T16:04:58.877 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:04:58.877 INFO:teuthology.orchestra.run.smithi169.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:04:58.878 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:04:58.878 INFO:teuthology.task.kernel:Updating grub on smithi169 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:04:58.878 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:04:59.552 INFO:teuthology.orchestra.run.smithi169.stderr:Generating grub configuration file ... 2024-03-25T16:05:00.903 INFO:teuthology.orchestra.run.smithi169.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:05:00.953 INFO:teuthology.orchestra.run.smithi169.stderr:done 2024-03-25T16:05:00.955 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-03-25T16:05:00.971 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.YcsKidYGpc 2024-03-25T16:05:00.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.YcsKidYGpc 2024-03-25T16:05:01.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.YcsKidYGpc 2024-03-25T16:05:01.177 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.YcsKidYGpc is 6KB 2024-03-25T16:05:01.225 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.YcsKidYGpc 2024-03-25T16:05:01.240 DEBUG:teuthology.orchestra.run.smithi169:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:05:01.313 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:05:01.313 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:05:01.313 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:05:01.313 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:05:01.314 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:05:01.465 DEBUG:teuthology.orchestra.run.smithi169:> sudo shutdown -r now 2024-03-25T16:05:25.428 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:05:25.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-03-25T16:05:25.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:05:31.470 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:05:31.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-25T16:05:31.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:06:06.111 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-25T16:06:15.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-25T16:06:15.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:06:25.442 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:06:33.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-03-25T16:06:34.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-03-25T16:06:34.446 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:06:34.863 DEBUG:teuthology.orchestra.run.smithi018:> true 2024-03-25T16:06:35.312 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-03-25T16:06:35.312 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:06:35.312 DEBUG:teuthology.orchestra.run.smithi018:> uname -r 2024-03-25T16:06:35.326 INFO:teuthology.orchestra.run.smithi018.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:06:35.326 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:06:35.326 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:06:35.326 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:06:36.327 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:06:36.327 DEBUG:teuthology.orchestra.run.smithi018:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:06:36.431 INFO:teuthology.orchestra.run.smithi018.stdout:ttyS1 2024-03-25T16:06:36.488 DEBUG:teuthology.parallel:result is None 2024-03-25T16:06:45.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-25T16:06:45.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:06:45.930 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-03-25T16:06:46.360 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-25T16:06:46.360 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:06:46.360 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-03-25T16:06:46.375 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:06:46.375 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:06:46.375 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:06:46.375 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:06:47.376 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:06:47.377 DEBUG:teuthology.orchestra.run.smithi169:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:06:47.435 INFO:teuthology.orchestra.run.smithi169.stdout:ttyS1 2024-03-25T16:06:47.473 DEBUG:teuthology.parallel:result is None 2024-03-25T16:06:47.474 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:06:47.483 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:06:47.483 DEBUG:teuthology.orchestra.run.smithi018:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:06:47.487 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:06:47.503 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:06:47.521 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:06:47.547 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:06:47.547 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:06:47.552 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:06:47.680 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:06:47.689 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:06:47.690 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:06:47.692 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:06:47.722 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:06:47.731 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:06:47.736 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:06:47.745 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:06:47.746 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:06:47.758 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:06:47.758 DEBUG:teuthology.orchestra.run.smithi018:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:06:47.789 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:06:47.817 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:06:47.830 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:06:47.830 DEBUG:teuthology.orchestra.run.smithi018:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:06:47.860 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:06:47.875 DEBUG:teuthology.orchestra.run.smithi018:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:06:47.958 DEBUG:teuthology.orchestra.run.smithi018:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:06:48.009 DEBUG:teuthology.orchestra.run.smithi018:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:06:48.096 DEBUG:teuthology.orchestra.run.smithi018:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:06:48.138 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T16:06:48.138 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:06:48.210 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:06:48.253 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:06:48.301 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:06:48.383 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:06:48.425 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T16:06:48.425 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:06:48.493 DEBUG:teuthology.orchestra.run.smithi018:> sudo service rsyslog restart 2024-03-25T16:06:48.495 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2024-03-25T16:06:48.543 INFO:teuthology.orchestra.run.smithi018.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:06:48.579 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:06:48.976 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:06:48.987 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:06:48.987 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:06:49.016 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:06:49.045 DEBUG:teuthology.orchestra.run.smithi018:> sudo service auditd rotate 2024-03-25T16:06:49.088 INFO:teuthology.orchestra.run.smithi018.stdout:Rotating logs: 2024-03-25T16:06:49.090 DEBUG:teuthology.orchestra.run.smithi169:> sudo service auditd rotate 2024-03-25T16:06:49.141 INFO:teuthology.orchestra.run.smithi169.stdout:Rotating logs: 2024-03-25T16:06:49.143 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:06:49.143 DEBUG:teuthology.orchestra.run.smithi018:> /usr/sbin/getenforce 2024-03-25T16:06:49.183 INFO:teuthology.orchestra.run.smithi018.stdout:Permissive 2024-03-25T16:06:49.183 DEBUG:teuthology.orchestra.run.smithi169:> /usr/sbin/getenforce 2024-03-25T16:06:49.205 INFO:teuthology.orchestra.run.smithi169.stdout:Permissive 2024-03-25T16:06:49.205 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi018.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi169.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:06:49.205 DEBUG:teuthology.orchestra.run.smithi018:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:06:49.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:06:49.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:06:49.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:06:49.282 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:06:49.283 DEBUG:teuthology.orchestra.run.smithi018:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:06:49.327 DEBUG:teuthology.orchestra.run.smithi169:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:06:49.366 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:06:49.379 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:06:49.399 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:06:49.401 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi018.front.sepia.ceph.com,smithi169.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:11:12.262 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi018.front.sepia.ceph.com'), Remote(name='ubuntu@smithi169.front.sepia.ceph.com')] 2024-03-25T16:11:12.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-03-25T16:11:12.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:11:12.387 DEBUG:teuthology.orchestra.run.smithi018:> true 2024-03-25T16:11:12.481 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-03-25T16:11:12.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-25T16:11:12.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:11:12.577 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-03-25T16:11:12.654 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-03-25T16:11:12.654 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:11:12.671 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:11:12.672 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:11:12.672 DEBUG:teuthology.orchestra.run.smithi018:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:11:12.675 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:11:12.676 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:11:12.706 INFO:teuthology.orchestra.run.smithi018.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:11:12.723 INFO:teuthology.orchestra.run.smithi018.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:11:12.726 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:11:12.744 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:11:12.753 INFO:teuthology.orchestra.run.smithi018.stderr:sudo: ntpd: command not found 2024-03-25T16:11:12.767 INFO:teuthology.orchestra.run.smithi018.stdout:506 Cannot talk to daemon 2024-03-25T16:11:12.773 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: ntpd: command not found 2024-03-25T16:11:12.784 INFO:teuthology.orchestra.run.smithi018.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:11:12.787 INFO:teuthology.orchestra.run.smithi169.stdout:506 Cannot talk to daemon 2024-03-25T16:11:12.801 INFO:teuthology.orchestra.run.smithi018.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:11:12.804 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:11:12.820 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:11:12.852 INFO:teuthology.orchestra.run.smithi018.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:11:12.855 INFO:teuthology.orchestra.run.smithi018.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:11:12.855 INFO:teuthology.orchestra.run.smithi018.stdout:=============================================================================== 2024-03-25T16:11:12.855 INFO:teuthology.orchestra.run.smithi018.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:11:12.855 INFO:teuthology.orchestra.run.smithi018.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:11:12.855 INFO:teuthology.orchestra.run.smithi018.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:11:12.856 INFO:teuthology.orchestra.run.smithi018.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:11:12.879 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:11:12.882 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:11:12.882 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-03-25T16:11:12.883 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:11:12.883 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:11:12.883 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:11:12.883 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:11:12.884 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:11:12.892 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:11:12.892 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:11:12.892 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:11:12.892 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:11:12.899 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:11:12.899 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:11:12.899 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:11:12.900 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:11:12.902 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:11:12.902 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:11:13.160 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:11:13.161 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:11:13.182 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:11:13.182 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:11:13.299 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:11:13.299 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T16:11:13.300 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:11:13.336 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:11:13.507 DEBUG:teuthology.orchestra.run.smithi169:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:11:13.510 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:11:13.510 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T16:11:13.510 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:11:13.536 DEBUG:teuthology.orchestra.run.smithi169:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:11:13.540 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:11:13.540 DEBUG:teuthology.orchestra.run.smithi018:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:11:13.610 DEBUG:teuthology.orchestra.run.smithi018:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:11:13.617 DEBUG:teuthology.orchestra.run.smithi169:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:11:13.683 INFO:teuthology.orchestra.run.smithi169.stdout:check_obsoletes = 1 2024-03-25T16:11:13.685 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum clean all 2024-03-25T16:11:13.692 DEBUG:teuthology.orchestra.run.smithi018:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:11:13.759 INFO:teuthology.orchestra.run.smithi018.stdout:check_obsoletes = 1 2024-03-25T16:11:13.760 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum clean all 2024-03-25T16:11:13.932 INFO:teuthology.orchestra.run.smithi169.stdout:74 files removed 2024-03-25T16:11:13.960 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-radosgw 2024-03-25T16:11:14.009 INFO:teuthology.orchestra.run.smithi018.stdout:74 files removed 2024-03-25T16:11:14.037 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-radosgw 2024-03-25T16:11:14.692 INFO:teuthology.orchestra.run.smithi169.stdout:ceph packages for x86_64 190 kB/s | 82 kB 00:00 2024-03-25T16:11:14.714 INFO:teuthology.orchestra.run.smithi018.stdout:ceph packages for x86_64 222 kB/s | 82 kB 00:00 2024-03-25T16:11:15.168 INFO:teuthology.orchestra.run.smithi169.stdout:ceph noarch packages 23 kB/s | 10 kB 00:00 2024-03-25T16:11:15.342 INFO:teuthology.orchestra.run.smithi018.stdout:ceph noarch packages 17 kB/s | 10 kB 00:00 2024-03-25T16:11:15.485 INFO:teuthology.orchestra.run.smithi169.stdout:ceph source packages 6.8 kB/s | 1.9 kB 00:00 2024-03-25T16:11:15.640 INFO:teuthology.orchestra.run.smithi018.stdout:ceph source packages 7.3 kB/s | 1.9 kB 00:00 2024-03-25T16:11:16.365 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - AppStream 8.0 MB/s | 6.7 MB 00:00 2024-03-25T16:11:16.393 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS-9 - AppStream 9.4 MB/s | 6.7 MB 00:00 2024-03-25T16:11:18.068 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-25T16:11:18.079 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-03-25T16:11:19.112 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-25T16:11:20.245 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.0 MB 00:01 2024-03-25T16:11:20.980 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-25T16:11:22.628 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-25T16:11:25.056 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-25T16:11:26.490 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-03-25T16:11:27.429 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 6.1 MB 00:01 2024-03-25T16:11:29.027 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-03-25T16:11:40.403 INFO:teuthology.orchestra.run.smithi018.stdout:Extra Packages for Enterprise Linux 1.8 MB/s | 21 MB 00:11 2024-03-25T16:11:45.535 INFO:teuthology.orchestra.run.smithi018.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-03-25T16:11:48.086 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:11:48.096 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:11:48.097 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T16:11:48.097 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:11:48.097 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:11:48.097 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:11:48.097 INFO:teuthology.orchestra.run.smithi018.stdout:Upgrading: 2024-03-25T16:11:48.097 INFO:teuthology.orchestra.run.smithi018.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:11:48.097 INFO:teuthology.orchestra.run.smithi018.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:11:48.097 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T16:11:48.097 INFO:teuthology.orchestra.run.smithi018.stdout: boost-program-options 2024-03-25T16:11:48.097 INFO:teuthology.orchestra.run.smithi018.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:11:48.097 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:11:48.098 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:11:48.098 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:11:48.098 INFO:teuthology.orchestra.run.smithi018.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:11:48.098 INFO:teuthology.orchestra.run.smithi018.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:11:48.098 INFO:teuthology.orchestra.run.smithi018.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:11:48.098 INFO:teuthology.orchestra.run.smithi018.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:11:48.098 INFO:teuthology.orchestra.run.smithi018.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:11:48.098 INFO:teuthology.orchestra.run.smithi018.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:11:48.098 INFO:teuthology.orchestra.run.smithi018.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:11:48.099 INFO:teuthology.orchestra.run.smithi018.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:11:48.099 INFO:teuthology.orchestra.run.smithi018.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:11:48.099 INFO:teuthology.orchestra.run.smithi018.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:11:48.099 INFO:teuthology.orchestra.run.smithi018.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:11:48.099 INFO:teuthology.orchestra.run.smithi018.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:11:48.099 INFO:teuthology.orchestra.run.smithi018.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:11:48.099 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ceph-argparse 2024-03-25T16:11:48.099 INFO:teuthology.orchestra.run.smithi018.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:11:48.099 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:11:48.099 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:11:48.099 INFO:teuthology.orchestra.run.smithi018.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:11:48.099 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:11:48.099 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:11:48.099 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:11:48.100 INFO:teuthology.orchestra.run.smithi018.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:11:48.100 INFO:teuthology.orchestra.run.smithi018.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:11:48.100 INFO:teuthology.orchestra.run.smithi018.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:11:48.100 INFO:teuthology.orchestra.run.smithi018.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:11:48.100 INFO:teuthology.orchestra.run.smithi018.stdout:Installing weak dependencies: 2024-03-25T16:11:48.100 INFO:teuthology.orchestra.run.smithi018.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:11:48.100 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:11:48.100 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:11:48.100 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:11:48.100 INFO:teuthology.orchestra.run.smithi018.stdout:Install 30 Packages 2024-03-25T16:11:48.100 INFO:teuthology.orchestra.run.smithi018.stdout:Upgrade 2 Packages 2024-03-25T16:11:48.100 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:11:48.101 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 58 M 2024-03-25T16:11:48.101 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:11:53.594 INFO:teuthology.orchestra.run.smithi018.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 9.7 MB 00:05 2024-03-25T16:11:53.661 INFO:teuthology.orchestra.run.smithi018.stdout:(2/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.0 MB/s | 5.3 MB 00:05 2024-03-25T16:11:53.845 INFO:teuthology.orchestra.run.smithi018.stdout:(3/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 3.7 MB/s | 20 MB 00:05 2024-03-25T16:11:53.862 INFO:teuthology.orchestra.run.smithi018.stdout:(4/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 94 kB/s | 25 kB 00:00 2024-03-25T16:11:54.170 INFO:teuthology.orchestra.run.smithi018.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.5 MB/s | 498 kB 00:00 2024-03-25T16:11:54.262 INFO:teuthology.orchestra.run.smithi018.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 492 kB/s | 45 kB 00:00 2024-03-25T16:11:54.362 INFO:teuthology.orchestra.run.smithi018.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.3 MB/s | 131 kB 00:00 2024-03-25T16:11:54.571 INFO:teuthology.orchestra.run.smithi018.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 777 kB/s | 162 kB 00:00 2024-03-25T16:11:54.763 INFO:teuthology.orchestra.run.smithi018.stdout:(9/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 673 kB/s | 742 kB 00:01 2024-03-25T16:11:54.788 INFO:teuthology.orchestra.run.smithi018.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 1.4 MB/s | 320 kB 00:00 2024-03-25T16:11:54.864 INFO:teuthology.orchestra.run.smithi018.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.3 MB/s | 99 kB 00:00 2024-03-25T16:11:55.022 INFO:teuthology.orchestra.run.smithi018.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 668 kB/s | 106 kB 00:00 2024-03-25T16:11:55.097 INFO:teuthology.orchestra.run.smithi018.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-03-25T16:11:55.139 INFO:teuthology.orchestra.run.smithi018.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T16:11:55.223 INFO:teuthology.orchestra.run.smithi018.stdout:(15/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 648 kB/s | 298 kB 00:00 2024-03-25T16:11:55.248 INFO:teuthology.orchestra.run.smithi018.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2024-03-25T16:11:55.290 INFO:teuthology.orchestra.run.smithi018.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-25T16:11:55.315 INFO:teuthology.orchestra.run.smithi018.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 615 kB/s | 41 kB 00:00 2024-03-25T16:11:55.482 INFO:teuthology.orchestra.run.smithi018.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 209 kB/s | 35 kB 00:00 2024-03-25T16:11:55.591 INFO:teuthology.orchestra.run.smithi018.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 412 kB/s | 44 kB 00:00 2024-03-25T16:11:55.616 INFO:teuthology.orchestra.run.smithi018.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 665 kB/s | 216 kB 00:00 2024-03-25T16:11:55.916 INFO:teuthology.orchestra.run.smithi018.stdout:(22/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 2.5 MB/s | 5.1 MB 00:02 2024-03-25T16:11:56.192 INFO:teuthology.orchestra.run.smithi018.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 514 kB/s | 309 kB 00:00 2024-03-25T16:11:56.217 INFO:teuthology.orchestra.run.smithi018.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 86 kB/s | 26 kB 00:00 2024-03-25T16:11:56.292 INFO:teuthology.orchestra.run.smithi018.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-03-25T16:11:56.401 INFO:teuthology.orchestra.run.smithi018.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 367 kB/s | 67 kB 00:00 2024-03-25T16:11:56.601 INFO:teuthology.orchestra.run.smithi018.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 959 kB/s | 192 kB 00:00 2024-03-25T16:11:56.776 INFO:teuthology.orchestra.run.smithi018.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.7 MB/s | 838 kB 00:00 2024-03-25T16:11:57.076 INFO:teuthology.orchestra.run.smithi018.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 4.4 MB 00:01 2024-03-25T16:11:57.468 INFO:teuthology.orchestra.run.smithi018.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 1.8 MB/s | 1.6 MB 00:00 2024-03-25T16:11:58.510 INFO:teuthology.orchestra.run.smithi018.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 2.2 MB/s | 3.1 MB 00:01 2024-03-25T16:12:00.985 INFO:teuthology.orchestra.run.smithi018.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 821 kB/s | 3.4 MB 00:04 2024-03-25T16:12:00.988 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:00.988 INFO:teuthology.orchestra.run.smithi018.stdout:Total 4.5 MB/s | 58 MB 00:12 2024-03-25T16:12:01.022 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:12:01.064 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:12:01.064 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:12:01.554 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:12:01.554 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:12:03.075 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:12:03.154 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:12:03.181 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:12:03.212 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:12:03.238 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:12:03.254 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:12:03.450 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:12:03.503 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:12:03.848 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:12:03.900 INFO:teuthology.orchestra.run.smithi018.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:12:03.993 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:12:04.082 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:12:04.109 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:12:04.180 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:12:04.209 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:12:04.254 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:12:04.324 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:12:04.345 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:12:04.391 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:12:04.418 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:12:04.531 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:12:04.570 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:12:04.610 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:12:04.685 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:12:04.705 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:12:04.729 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:12:05.078 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:12:05.110 INFO:teuthology.orchestra.run.smithi018.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:12:05.200 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:12:05.231 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:12:05.276 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:12:05.756 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:12:05.787 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:12:05.858 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:12:05.929 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:12:05.956 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:12:08.210 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:12:08.243 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:12:08.815 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:12:08.876 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:12:08.972 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:12:08.973 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:12:08.973 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:12:08.973 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:09.015 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:12:18.903 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:12:18.904 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /sys 2024-03-25T16:12:18.904 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /proc 2024-03-25T16:12:18.904 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /mnt 2024-03-25T16:12:18.904 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /var/tmp 2024-03-25T16:12:18.904 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /home 2024-03-25T16:12:18.904 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /root 2024-03-25T16:12:18.904 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /tmp 2024-03-25T16:12:18.904 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:19.831 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:12:19.855 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:12:19.855 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:19.855 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:12:19.855 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:12:19.855 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:12:19.856 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:19.857 INFO:teuthology.orchestra.run.smithi018.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:12:19.950 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:12:19.951 INFO:teuthology.orchestra.run.smithi018.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:12:21.101 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:12:21.102 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:12:21.102 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:12:21.102 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:12:21.102 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:12:21.102 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:12:21.102 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:12:21.102 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:12:21.102 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:12:21.102 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:12:21.102 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:12:21.102 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:12:21.103 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:12:21.103 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:12:21.103 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:12:21.103 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:12:21.103 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:12:21.103 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:12:21.103 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:12:21.103 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:12:21.103 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:12:21.103 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:12:21.103 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:12:21.104 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:12:21.104 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:12:21.104 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:12:21.104 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:12:21.104 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:12:21.104 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:12:21.104 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:12:21.104 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:12:21.104 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:12:21.104 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:12:21.104 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:12:22.150 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:12:22.150 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:22.150 INFO:teuthology.orchestra.run.smithi018.stdout:Upgraded: 2024-03-25T16:12:22.150 INFO:teuthology.orchestra.run.smithi018.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:22.150 INFO:teuthology.orchestra.run.smithi018.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:22.151 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:12:22.151 INFO:teuthology.orchestra.run.smithi018.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:12:22.151 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:22.151 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:22.151 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:22.151 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:22.151 INFO:teuthology.orchestra.run.smithi018.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:12:22.151 INFO:teuthology.orchestra.run.smithi018.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:12:22.151 INFO:teuthology.orchestra.run.smithi018.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:12:22.151 INFO:teuthology.orchestra.run.smithi018.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:22.151 INFO:teuthology.orchestra.run.smithi018.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:12:22.151 INFO:teuthology.orchestra.run.smithi018.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:12:22.152 INFO:teuthology.orchestra.run.smithi018.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:12:22.152 INFO:teuthology.orchestra.run.smithi018.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:22.152 INFO:teuthology.orchestra.run.smithi018.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:12:22.152 INFO:teuthology.orchestra.run.smithi018.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:22.152 INFO:teuthology.orchestra.run.smithi018.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:12:22.152 INFO:teuthology.orchestra.run.smithi018.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:12:22.152 INFO:teuthology.orchestra.run.smithi018.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:12:22.152 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:22.152 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:22.152 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:22.152 INFO:teuthology.orchestra.run.smithi018.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:12:22.152 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:22.152 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:22.153 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:22.153 INFO:teuthology.orchestra.run.smithi018.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:12:22.153 INFO:teuthology.orchestra.run.smithi018.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:12:22.153 INFO:teuthology.orchestra.run.smithi018.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:12:22.153 INFO:teuthology.orchestra.run.smithi018.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:12:22.153 INFO:teuthology.orchestra.run.smithi018.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:12:22.153 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:22.153 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:12:22.505 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-test 2024-03-25T16:12:23.085 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:00:38 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:12:23.249 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:12:23.250 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:12:23.250 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T16:12:23.250 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:12:23.251 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:12:23.251 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:12:23.251 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T16:12:23.251 INFO:teuthology.orchestra.run.smithi018.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:12:23.251 INFO:teuthology.orchestra.run.smithi018.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:12:23.251 INFO:teuthology.orchestra.run.smithi018.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:12:23.251 INFO:teuthology.orchestra.run.smithi018.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:12:23.251 INFO:teuthology.orchestra.run.smithi018.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:12:23.251 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:23.251 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:12:23.251 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:12:23.251 INFO:teuthology.orchestra.run.smithi018.stdout:Install 6 Packages 2024-03-25T16:12:23.251 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:23.252 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 48 M 2024-03-25T16:12:23.252 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 204 M 2024-03-25T16:12:23.252 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:12:23.721 INFO:teuthology.orchestra.run.smithi018.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-25T16:12:23.779 INFO:teuthology.orchestra.run.smithi018.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-25T16:12:23.813 INFO:teuthology.orchestra.run.smithi018.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T16:12:23.980 INFO:teuthology.orchestra.run.smithi018.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 361 kB/s | 163 kB 00:00 2024-03-25T16:12:24.722 INFO:teuthology.orchestra.run.smithi018.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 206 kB/s | 187 kB 00:00 2024-03-25T16:12:31.204 INFO:teuthology.orchestra.run.smithi018.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 6.2 MB/s | 47 MB 00:07 2024-03-25T16:12:31.207 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:31.208 INFO:teuthology.orchestra.run.smithi018.stdout:Total 6.1 MB/s | 48 MB 00:07 2024-03-25T16:12:31.221 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:12:31.245 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:12:31.245 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:12:31.638 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:12:31.639 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:12:32.229 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:12:32.299 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:12:32.349 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:12:32.389 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:12:32.437 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:12:32.468 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:12:37.200 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:12:37.278 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:12:39.577 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:12:39.577 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:12:39.577 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:12:39.577 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:12:39.577 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:12:39.577 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:12:39.923 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:12:39.923 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:39.923 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:12:39.923 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:39.923 INFO:teuthology.orchestra.run.smithi018.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:12:39.923 INFO:teuthology.orchestra.run.smithi018.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:39.923 INFO:teuthology.orchestra.run.smithi018.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:12:39.923 INFO:teuthology.orchestra.run.smithi018.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:12:39.923 INFO:teuthology.orchestra.run.smithi018.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:12:39.924 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:39.924 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:12:40.208 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph 2024-03-25T16:12:40.784 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:00:55 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:12:40.944 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:12:40.946 INFO:teuthology.orchestra.run.smithi018.stdout:========================================================================================= 2024-03-25T16:12:40.946 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T16:12:40.947 INFO:teuthology.orchestra.run.smithi018.stdout:========================================================================================= 2024-03-25T16:12:40.947 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:12:40.947 INFO:teuthology.orchestra.run.smithi018.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:12:40.947 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T16:12:40.947 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:12:40.947 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:12:40.947 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:12:40.947 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:12:40.947 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:12:40.947 INFO:teuthology.orchestra.run.smithi018.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:12:40.947 INFO:teuthology.orchestra.run.smithi018.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:12:40.947 INFO:teuthology.orchestra.run.smithi018.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:12:40.947 INFO:teuthology.orchestra.run.smithi018.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:12:40.948 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:12:40.948 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:12:40.948 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:12:40.948 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:12:40.948 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:12:40.948 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:12:40.948 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:12:40.948 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:12:40.948 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:12:40.948 INFO:teuthology.orchestra.run.smithi018.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:12:40.948 INFO:teuthology.orchestra.run.smithi018.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:12:40.948 INFO:teuthology.orchestra.run.smithi018.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:12:40.948 INFO:teuthology.orchestra.run.smithi018.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:12:40.949 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:12:40.949 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:12:40.949 INFO:teuthology.orchestra.run.smithi018.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:12:40.949 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:12:40.949 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:12:40.949 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:12:40.949 INFO:teuthology.orchestra.run.smithi018.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:12:40.949 INFO:teuthology.orchestra.run.smithi018.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:12:40.949 INFO:teuthology.orchestra.run.smithi018.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:12:40.949 INFO:teuthology.orchestra.run.smithi018.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:12:40.949 INFO:teuthology.orchestra.run.smithi018.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:12:40.949 INFO:teuthology.orchestra.run.smithi018.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:12:40.949 INFO:teuthology.orchestra.run.smithi018.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:12:40.950 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:40.950 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:12:40.950 INFO:teuthology.orchestra.run.smithi018.stdout:========================================================================================= 2024-03-25T16:12:40.950 INFO:teuthology.orchestra.run.smithi018.stdout:Install 36 Packages 2024-03-25T16:12:40.950 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:40.950 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 30 M 2024-03-25T16:12:40.950 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 107 M 2024-03-25T16:12:40.951 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:12:41.509 INFO:teuthology.orchestra.run.smithi018.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 21 kB/s | 6.6 kB 00:00 2024-03-25T16:12:41.951 INFO:teuthology.orchestra.run.smithi018.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.0 MB/s | 1.5 MB 00:00 2024-03-25T16:12:42.760 INFO:teuthology.orchestra.run.smithi018.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.4 MB/s | 2.1 MB 00:01 2024-03-25T16:12:42.926 INFO:teuthology.orchestra.run.smithi018.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.5 MB/s | 247 kB 00:00 2024-03-25T16:12:43.043 INFO:teuthology.orchestra.run.smithi018.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T16:12:43.077 INFO:teuthology.orchestra.run.smithi018.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-25T16:12:43.111 INFO:teuthology.orchestra.run.smithi018.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 945 kB/s | 32 kB 00:00 2024-03-25T16:12:43.144 INFO:teuthology.orchestra.run.smithi018.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:12:43.178 INFO:teuthology.orchestra.run.smithi018.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:12:43.228 INFO:teuthology.orchestra.run.smithi018.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T16:12:43.312 INFO:teuthology.orchestra.run.smithi018.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-25T16:12:43.354 INFO:teuthology.orchestra.run.smithi018.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T16:12:43.396 INFO:teuthology.orchestra.run.smithi018.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T16:12:43.438 INFO:teuthology.orchestra.run.smithi018.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:12:43.554 INFO:teuthology.orchestra.run.smithi018.stdout:(15/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 2.3 MB/s | 4.7 MB 00:02 2024-03-25T16:12:43.605 INFO:teuthology.orchestra.run.smithi018.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 749 kB/s | 125 kB 00:00 2024-03-25T16:12:44.006 INFO:teuthology.orchestra.run.smithi018.stdout:(17/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 8.2 MB/s | 17 MB 00:02 2024-03-25T16:12:44.039 INFO:teuthology.orchestra.run.smithi018.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 348 kB/s | 151 kB 00:00 2024-03-25T16:12:44.131 INFO:teuthology.orchestra.run.smithi018.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-03-25T16:12:44.148 INFO:teuthology.orchestra.run.smithi018.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 309 kB/s | 43 kB 00:00 2024-03-25T16:12:44.198 INFO:teuthology.orchestra.run.smithi018.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-03-25T16:12:44.223 INFO:teuthology.orchestra.run.smithi018.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-03-25T16:12:44.249 INFO:teuthology.orchestra.run.smithi018.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-03-25T16:12:44.265 INFO:teuthology.orchestra.run.smithi018.stdout:(24/36): lua-devel-5.4.4-4.el9.x86_64.rpm 31 kB/s | 22 kB 00:00 2024-03-25T16:12:44.291 INFO:teuthology.orchestra.run.smithi018.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 343 kB/s | 23 kB 00:00 2024-03-25T16:12:44.316 INFO:teuthology.orchestra.run.smithi018.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 294 kB/s | 19 kB 00:00 2024-03-25T16:12:44.350 INFO:teuthology.orchestra.run.smithi018.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 790 kB/s | 46 kB 00:00 2024-03-25T16:12:44.400 INFO:teuthology.orchestra.run.smithi018.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 941 kB/s | 79 kB 00:00 2024-03-25T16:12:44.425 INFO:teuthology.orchestra.run.smithi018.stdout:(29/36): python3-jaraco-text-3.2.0-6.el9.noarch 123 kB/s | 20 kB 00:00 2024-03-25T16:12:44.459 INFO:teuthology.orchestra.run.smithi018.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-25T16:12:44.484 INFO:teuthology.orchestra.run.smithi018.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-03-25T16:12:44.518 INFO:teuthology.orchestra.run.smithi018.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 611 kB/s | 36 kB 00:00 2024-03-25T16:12:44.543 INFO:teuthology.orchestra.run.smithi018.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 766 kB/s | 90 kB 00:00 2024-03-25T16:12:44.593 INFO:teuthology.orchestra.run.smithi018.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-03-25T16:12:44.618 INFO:teuthology.orchestra.run.smithi018.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-03-25T16:12:44.644 INFO:teuthology.orchestra.run.smithi018.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-03-25T16:12:44.647 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:44.647 INFO:teuthology.orchestra.run.smithi018.stdout:Total 8.0 MB/s | 30 MB 00:03 2024-03-25T16:12:44.672 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:12:44.718 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:12:44.719 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:12:45.151 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:12:45.151 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:12:46.523 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:12:46.580 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:12:46.613 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:12:46.640 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:12:46.714 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:12:46.767 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:12:46.798 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:12:46.893 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:12:46.969 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:12:47.026 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:12:47.100 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:12:47.152 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:12:47.189 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:12:47.248 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:12:47.312 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:12:47.510 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:12:47.561 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:12:47.607 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:12:47.642 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:12:47.672 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:12:47.752 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:12:47.782 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:12:47.823 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:12:47.864 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:12:47.908 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:12:47.950 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:12:48.044 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:12:48.081 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:12:48.158 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:12:48.309 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:12:48.446 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:12:48.614 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:12:48.652 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:12:48.677 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:12:48.678 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:48.678 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:12:48.678 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:12:48.678 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:12:48.678 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:50.195 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:12:50.221 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:12:50.222 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:50.222 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:12:50.222 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:12:50.222 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:12:50.222 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:50.696 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:12:50.719 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:12:50.719 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:50.719 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:12:50.719 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:12:50.719 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:12:50.719 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:50.938 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:12:50.961 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:12:50.961 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:50.962 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:12:50.962 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:12:50.962 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:12:50.962 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:51.030 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:12:52.799 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:12:52.799 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:12:52.799 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:12:52.799 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:12:52.799 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:12:52.799 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:12:52.799 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:12:52.799 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:12:52.799 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:12:52.799 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:12:52.799 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:12:52.799 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:12:52.800 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:12:52.800 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:12:52.800 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:12:52.800 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:12:52.800 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:12:52.800 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:12:52.800 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:12:52.800 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:12:52.800 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:12:52.800 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:12:52.800 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:12:52.800 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:12:52.800 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:12:52.800 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:12:52.801 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:12:52.801 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:12:52.801 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:12:52.801 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:12:52.801 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:12:52.801 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:12:52.801 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:12:52.801 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:12:52.801 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:12:52.801 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:12:53.932 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:12:53.933 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:53.933 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:12:53.933 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:53.933 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:53.933 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:53.933 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:53.933 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:53.933 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:53.933 INFO:teuthology.orchestra.run.smithi018.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:12:53.933 INFO:teuthology.orchestra.run.smithi018.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:12:53.933 INFO:teuthology.orchestra.run.smithi018.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:12:53.934 INFO:teuthology.orchestra.run.smithi018.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:12:53.934 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:12:53.934 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:12:53.934 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:12:53.934 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:12:53.934 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:12:53.934 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:12:53.934 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:12:53.934 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:12:53.934 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:12:53.934 INFO:teuthology.orchestra.run.smithi018.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:12:53.934 INFO:teuthology.orchestra.run.smithi018.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:12:53.934 INFO:teuthology.orchestra.run.smithi018.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:12:53.934 INFO:teuthology.orchestra.run.smithi018.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:12:53.935 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:12:53.935 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:12:53.935 INFO:teuthology.orchestra.run.smithi018.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:12:53.935 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:12:53.935 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:12:53.935 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:12:53.935 INFO:teuthology.orchestra.run.smithi018.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:12:53.935 INFO:teuthology.orchestra.run.smithi018.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:12:53.935 INFO:teuthology.orchestra.run.smithi018.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:12:53.935 INFO:teuthology.orchestra.run.smithi018.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:12:53.935 INFO:teuthology.orchestra.run.smithi018.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:12:53.935 INFO:teuthology.orchestra.run.smithi018.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:12:53.935 INFO:teuthology.orchestra.run.smithi018.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:12:53.935 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:53.936 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:12:54.233 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-base 2024-03-25T16:12:54.810 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:09 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:12:54.910 INFO:teuthology.orchestra.run.smithi018.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:12:54.968 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:12:54.969 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T16:12:54.970 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:12:55.042 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install cephadm 2024-03-25T16:12:55.611 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:10 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:12:55.766 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:12:55.767 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:12:55.767 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T16:12:55.767 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:12:55.768 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:12:55.768 INFO:teuthology.orchestra.run.smithi018.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:12:55.768 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:55.768 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:12:55.768 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:12:55.768 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T16:12:55.768 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:55.768 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 747 k 2024-03-25T16:12:55.768 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 754 k 2024-03-25T16:12:55.768 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:12:56.355 INFO:teuthology.orchestra.run.smithi018.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.2 MB/s | 747 kB 00:00 2024-03-25T16:12:56.356 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:56.356 INFO:teuthology.orchestra.run.smithi018.stdout:Total 1.2 MB/s | 747 kB 00:00 2024-03-25T16:12:56.356 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:12:56.362 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:12:56.362 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:12:56.368 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:12:56.368 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:12:56.441 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:12:56.889 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:12:56.979 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:12:57.345 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:12:57.633 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:12:57.633 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:57.633 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:12:57.633 INFO:teuthology.orchestra.run.smithi018.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:12:57.633 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:57.633 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:12:57.846 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:12:58.424 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:13 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:12:58.583 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:12:58.583 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:12:58.583 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T16:12:58.584 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:12:58.584 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:12:58.584 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:12:58.584 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:58.584 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:12:58.584 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:12:58.584 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T16:12:58.584 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:12:58.584 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 145 k 2024-03-25T16:12:58.584 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 436 k 2024-03-25T16:12:58.584 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:12:59.278 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 210 kB/s | 145 kB 00:00 2024-03-25T16:12:59.279 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:59.279 INFO:teuthology.orchestra.run.smithi018.stdout:Total 209 kB/s | 145 kB 00:00 2024-03-25T16:12:59.279 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:12:59.287 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:12:59.287 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:12:59.342 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:12:59.342 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:12:59.490 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:12:59.554 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:12:59.578 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:12:59.579 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:59.579 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:12:59.579 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:00.527 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:13:00.527 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:00.528 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:13:00.528 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:00.528 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:00.528 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:00.777 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mgr 2024-03-25T16:13:01.350 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:16 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:01.452 INFO:teuthology.orchestra.run.smithi018.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:01.509 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:01.510 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T16:13:01.510 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:01.580 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:13:02.148 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:17 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:02.303 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:02.304 INFO:teuthology.orchestra.run.smithi018.stdout:====================================================================================== 2024-03-25T16:13:02.304 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T16:13:02.304 INFO:teuthology.orchestra.run.smithi018.stdout:====================================================================================== 2024-03-25T16:13:02.304 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:13:02.304 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:13:02.304 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T16:13:02.304 INFO:teuthology.orchestra.run.smithi018.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:13:02.304 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:13:02.304 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:13:02.304 INFO:teuthology.orchestra.run.smithi018.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:13:02.304 INFO:teuthology.orchestra.run.smithi018.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:13:02.304 INFO:teuthology.orchestra.run.smithi018.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:13:02.305 INFO:teuthology.orchestra.run.smithi018.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:13:02.305 INFO:teuthology.orchestra.run.smithi018.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:13:02.305 INFO:teuthology.orchestra.run.smithi018.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:13:02.305 INFO:teuthology.orchestra.run.smithi018.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:13:02.305 INFO:teuthology.orchestra.run.smithi018.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:13:02.305 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:02.305 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:13:02.305 INFO:teuthology.orchestra.run.smithi018.stdout:====================================================================================== 2024-03-25T16:13:02.305 INFO:teuthology.orchestra.run.smithi018.stdout:Install 12 Packages 2024-03-25T16:13:02.305 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:02.306 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 6.9 M 2024-03-25T16:13:02.306 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 26 M 2024-03-25T16:13:02.306 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:13:02.915 INFO:teuthology.orchestra.run.smithi018.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 47 kB/s | 15 kB 00:00 2024-03-25T16:13:02.932 INFO:teuthology.orchestra.run.smithi018.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 92 kB/s | 31 kB 00:00 2024-03-25T16:13:03.085 INFO:teuthology.orchestra.run.smithi018.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-25T16:13:03.116 INFO:teuthology.orchestra.run.smithi018.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-25T16:13:03.808 INFO:teuthology.orchestra.run.smithi018.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 793 kB/s | 548 kB 00:00 2024-03-25T16:13:03.908 INFO:teuthology.orchestra.run.smithi018.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 194 kB/s | 19 kB 00:00 2024-03-25T16:13:04.008 INFO:teuthology.orchestra.run.smithi018.stdout:(7/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 1.3 MB/s | 1.8 MB 00:01 2024-03-25T16:13:04.192 INFO:teuthology.orchestra.run.smithi018.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 7.2 MB/s | 2.0 MB 00:00 2024-03-25T16:13:04.292 INFO:teuthology.orchestra.run.smithi018.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 311 kB/s | 31 kB 00:00 2024-03-25T16:13:04.393 INFO:teuthology.orchestra.run.smithi018.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-25T16:13:04.543 INFO:teuthology.orchestra.run.smithi018.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 271 kB/s | 144 kB 00:00 2024-03-25T16:13:04.760 INFO:teuthology.orchestra.run.smithi018.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 515 kB/s | 863 kB 00:01 2024-03-25T16:13:04.768 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:04.768 INFO:teuthology.orchestra.run.smithi018.stdout:Total 2.8 MB/s | 6.9 MB 00:02 2024-03-25T16:13:04.788 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:13:04.808 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:13:04.808 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:13:05.020 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:13:05.020 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:13:05.538 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:13:05.619 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:13:05.654 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:13:05.702 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:13:05.731 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:13:05.847 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:13:05.980 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:13:06.071 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:13:06.120 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:13:06.148 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:13:06.180 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:13:06.461 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:13:06.496 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:13:07.255 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:13:07.256 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:13:07.256 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:13:07.256 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:13:07.256 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:13:07.256 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:13:07.256 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:13:07.256 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:13:07.256 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:13:07.256 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:13:07.256 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:13:07.256 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:13:07.604 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:13:07.604 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:07.604 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:13:07.604 INFO:teuthology.orchestra.run.smithi018.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:13:07.604 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:07.604 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:07.604 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:07.605 INFO:teuthology.orchestra.run.smithi018.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:13:07.605 INFO:teuthology.orchestra.run.smithi018.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:13:07.605 INFO:teuthology.orchestra.run.smithi018.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:13:07.605 INFO:teuthology.orchestra.run.smithi018.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:13:07.605 INFO:teuthology.orchestra.run.smithi018.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:13:07.605 INFO:teuthology.orchestra.run.smithi018.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:13:07.605 INFO:teuthology.orchestra.run.smithi018.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:13:07.605 INFO:teuthology.orchestra.run.smithi018.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:13:07.605 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:07.605 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:07.838 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:13:08.428 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:23 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:08.588 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:08.589 INFO:teuthology.orchestra.run.smithi018.stdout:============================================================================================ 2024-03-25T16:13:08.589 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T16:13:08.589 INFO:teuthology.orchestra.run.smithi018.stdout:============================================================================================ 2024-03-25T16:13:08.589 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:13:08.589 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:13:08.589 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T16:13:08.589 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:13:08.590 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:13:08.590 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:13:08.590 INFO:teuthology.orchestra.run.smithi018.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:13:08.590 INFO:teuthology.orchestra.run.smithi018.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:13:08.590 INFO:teuthology.orchestra.run.smithi018.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:13:08.590 INFO:teuthology.orchestra.run.smithi018.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:13:08.590 INFO:teuthology.orchestra.run.smithi018.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:13:08.590 INFO:teuthology.orchestra.run.smithi018.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:13:08.590 INFO:teuthology.orchestra.run.smithi018.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:13:08.590 INFO:teuthology.orchestra.run.smithi018.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:13:08.590 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:08.591 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:13:08.591 INFO:teuthology.orchestra.run.smithi018.stdout:============================================================================================ 2024-03-25T16:13:08.591 INFO:teuthology.orchestra.run.smithi018.stdout:Install 12 Packages 2024-03-25T16:13:08.591 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:08.591 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 37 M 2024-03-25T16:13:08.592 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 209 M 2024-03-25T16:13:08.592 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:13:09.101 INFO:teuthology.orchestra.run.smithi018.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 126 kB/s | 32 kB 00:00 2024-03-25T16:13:09.184 INFO:teuthology.orchestra.run.smithi018.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 207 kB/s | 17 kB 00:00 2024-03-25T16:13:09.268 INFO:teuthology.orchestra.run.smithi018.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 409 kB/s | 34 kB 00:00 2024-03-25T16:13:09.476 INFO:teuthology.orchestra.run.smithi018.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.8 MB/s | 3.0 MB 00:00 2024-03-25T16:13:09.810 INFO:teuthology.orchestra.run.smithi018.stdout:(5/12): ceph-mgr-diskprediction-local-19.0.0-14 7.7 MB/s | 7.4 MB 00:00 2024-03-25T16:13:09.877 INFO:teuthology.orchestra.run.smithi018.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 7.8 MB/s | 4.7 MB 00:00 2024-03-25T16:13:09.944 INFO:teuthology.orchestra.run.smithi018.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 4.8 MB 00:00 2024-03-25T16:13:09.969 INFO:teuthology.orchestra.run.smithi018.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.4 MB/s | 217 kB 00:00 2024-03-25T16:13:10.294 INFO:teuthology.orchestra.run.smithi018.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-25T16:13:10.386 INFO:teuthology.orchestra.run.smithi018.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 472 kB/s | 195 kB 00:00 2024-03-25T16:13:10.478 INFO:teuthology.orchestra.run.smithi018.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-25T16:13:10.520 INFO:teuthology.orchestra.run.smithi018.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-03-25T16:13:10.522 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:10.522 INFO:teuthology.orchestra.run.smithi018.stdout:Total 19 MB/s | 37 MB 00:01 2024-03-25T16:13:10.563 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:13:10.586 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:13:10.587 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:13:10.972 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:13:10.972 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:13:11.700 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:13:11.785 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:13:11.816 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:13:11.916 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:13:11.953 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:13:12.139 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:13:12.168 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:13:12.253 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:13:12.950 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:13:13.068 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:13:14.399 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:13:15.481 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:13:15.509 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:13:17.990 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:13:17.990 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:13:17.990 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:13:17.990 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:13:17.990 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:13:17.990 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:13:17.990 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:13:17.990 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:13:17.990 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:13:17.991 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:13:17.991 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:13:17.991 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:13:18.445 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:13:18.445 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:18.445 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:13:18.445 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:18.445 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:13:18.446 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:13:18.446 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:13:18.446 INFO:teuthology.orchestra.run.smithi018.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:13:18.446 INFO:teuthology.orchestra.run.smithi018.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:13:18.446 INFO:teuthology.orchestra.run.smithi018.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:13:18.446 INFO:teuthology.orchestra.run.smithi018.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:13:18.446 INFO:teuthology.orchestra.run.smithi018.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:13:18.446 INFO:teuthology.orchestra.run.smithi018.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:13:18.446 INFO:teuthology.orchestra.run.smithi018.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:13:18.446 INFO:teuthology.orchestra.run.smithi018.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:13:18.446 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:18.446 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:18.788 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:13:19.373 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:34 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:19.531 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:19.532 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T16:13:19.532 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T16:13:19.532 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T16:13:19.532 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:13:19.532 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:13:19.532 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T16:13:19.532 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:13:19.532 INFO:teuthology.orchestra.run.smithi018.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:13:19.532 INFO:teuthology.orchestra.run.smithi018.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:13:19.533 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:13:19.533 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:13:19.533 INFO:teuthology.orchestra.run.smithi018.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:13:19.533 INFO:teuthology.orchestra.run.smithi018.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:13:19.533 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:13:19.533 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:13:19.533 INFO:teuthology.orchestra.run.smithi018.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:13:19.533 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:13:19.533 INFO:teuthology.orchestra.run.smithi018.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:13:19.533 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:19.533 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:13:19.533 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T16:13:19.533 INFO:teuthology.orchestra.run.smithi018.stdout:Install 13 Packages 2024-03-25T16:13:19.534 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:19.534 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 2.0 M 2024-03-25T16:13:19.534 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 26 M 2024-03-25T16:13:19.534 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:13:19.743 INFO:teuthology.orchestra.run.smithi018.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 132 kB/s | 18 kB 00:00 2024-03-25T16:13:19.768 INFO:teuthology.orchestra.run.smithi018.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 150 kB/s | 24 kB 00:00 2024-03-25T16:13:19.843 INFO:teuthology.orchestra.run.smithi018.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 174 kB 00:00 2024-03-25T16:13:19.860 INFO:teuthology.orchestra.run.smithi018.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-25T16:13:19.902 INFO:teuthology.orchestra.run.smithi018.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T16:13:19.927 INFO:teuthology.orchestra.run.smithi018.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 637 kB/s | 43 kB 00:00 2024-03-25T16:13:19.969 INFO:teuthology.orchestra.run.smithi018.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 144 kB/s | 52 kB 00:00 2024-03-25T16:13:20.245 INFO:teuthology.orchestra.run.smithi018.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 47 kB/s | 15 kB 00:00 2024-03-25T16:13:20.270 INFO:teuthology.orchestra.run.smithi018.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 88 kB/s | 32 kB 00:00 2024-03-25T16:13:20.420 INFO:teuthology.orchestra.run.smithi018.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 394 kB/s | 59 kB 00:00 2024-03-25T16:13:20.445 INFO:teuthology.orchestra.run.smithi018.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 426 kB/s | 203 kB 00:00 2024-03-25T16:13:20.546 INFO:teuthology.orchestra.run.smithi018.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 716 kB/s | 90 kB 00:00 2024-03-25T16:13:20.671 INFO:teuthology.orchestra.run.smithi018.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.4 MB/s | 1.0 MB 00:00 2024-03-25T16:13:20.676 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:20.676 INFO:teuthology.orchestra.run.smithi018.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-03-25T16:13:20.676 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:13:20.691 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:13:20.692 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:13:20.793 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:13:20.793 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:13:21.335 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:13:21.507 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:13:21.557 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:13:21.629 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:13:21.665 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:13:21.710 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:13:21.803 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:13:21.912 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:13:21.967 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:13:22.729 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:13:22.780 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:13:22.814 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:13:22.863 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:13:22.892 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:13:23.738 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:13:23.739 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:13:23.739 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:13:23.739 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:13:23.739 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:13:23.739 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:13:23.739 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:13:23.739 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:13:23.739 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:13:23.739 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:13:23.739 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:13:23.739 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:13:23.739 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:13:24.100 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:13:24.100 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:24.100 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:13:24.100 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:24.100 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:13:24.100 INFO:teuthology.orchestra.run.smithi018.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:13:24.100 INFO:teuthology.orchestra.run.smithi018.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:13:24.100 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:13:24.101 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:13:24.101 INFO:teuthology.orchestra.run.smithi018.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:13:24.101 INFO:teuthology.orchestra.run.smithi018.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:13:24.101 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:13:24.101 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:13:24.101 INFO:teuthology.orchestra.run.smithi018.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:13:24.101 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:13:24.101 INFO:teuthology.orchestra.run.smithi018.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:13:24.101 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:24.101 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:24.344 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:13:24.919 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:39 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:25.075 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:25.076 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T16:13:25.076 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T16:13:25.076 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T16:13:25.076 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:13:25.077 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:13:25.077 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T16:13:25.077 INFO:teuthology.orchestra.run.smithi018.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:13:25.077 INFO:teuthology.orchestra.run.smithi018.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:13:25.077 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:13:25.077 INFO:teuthology.orchestra.run.smithi018.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:13:25.077 INFO:teuthology.orchestra.run.smithi018.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:13:25.077 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:25.077 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:13:25.077 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T16:13:25.077 INFO:teuthology.orchestra.run.smithi018.stdout:Install 6 Packages 2024-03-25T16:13:25.077 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:25.078 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 6.8 M 2024-03-25T16:13:25.078 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 33 M 2024-03-25T16:13:25.078 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:13:25.739 INFO:teuthology.orchestra.run.smithi018.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-03-25T16:13:25.823 INFO:teuthology.orchestra.run.smithi018.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 608 kB/s | 138 kB 00:00 2024-03-25T16:13:25.973 INFO:teuthology.orchestra.run.smithi018.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 385 kB/s | 58 kB 00:00 2024-03-25T16:13:26.024 INFO:teuthology.orchestra.run.smithi018.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.9 MB/s | 548 kB 00:00 2024-03-25T16:13:26.049 INFO:teuthology.orchestra.run.smithi018.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 692 kB/s | 51 kB 00:00 2024-03-25T16:13:26.157 INFO:teuthology.orchestra.run.smithi018.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 5.8 MB 00:00 2024-03-25T16:13:26.159 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:26.159 INFO:teuthology.orchestra.run.smithi018.stdout:Total 6.3 MB/s | 6.8 MB 00:01 2024-03-25T16:13:26.171 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:13:26.182 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:13:26.182 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:13:26.273 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:13:26.273 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:13:26.575 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:13:26.754 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:13:26.801 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:13:27.350 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:13:27.438 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:13:27.499 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:13:27.529 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:13:28.316 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:13:28.317 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:13:28.317 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:13:28.317 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:13:28.317 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:13:28.317 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi018.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi018.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi018.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi018.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:28.838 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-fuse 2024-03-25T16:13:29.414 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:44 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:29.570 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:29.571 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:29.571 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T16:13:29.571 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:29.571 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:13:29.571 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:13:29.571 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T16:13:29.571 INFO:teuthology.orchestra.run.smithi018.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:13:29.572 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:29.572 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:13:29.572 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:29.572 INFO:teuthology.orchestra.run.smithi018.stdout:Install 2 Packages 2024-03-25T16:13:29.572 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:29.572 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 964 k 2024-03-25T16:13:29.572 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 2.8 M 2024-03-25T16:13:29.572 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:13:29.800 INFO:teuthology.orchestra.run.smithi018.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 344 kB/s | 79 kB 00:00 2024-03-25T16:13:30.159 INFO:teuthology.orchestra.run.smithi018.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.5 MB/s | 885 kB 00:00 2024-03-25T16:13:30.160 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:30.160 INFO:teuthology.orchestra.run.smithi018.stdout:Total 1.6 MB/s | 964 kB 00:00 2024-03-25T16:13:30.160 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:13:30.182 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:13:30.182 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:13:30.257 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:13:30.258 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:13:30.443 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:13:30.590 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:13:30.671 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:13:31.409 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:13:31.409 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:13:31.729 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:13:31.730 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:31.730 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:13:31.730 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:13:31.730 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:31.730 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:32.019 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-volume 2024-03-25T16:13:32.603 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:47 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:32.763 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:32.763 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:32.763 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-25T16:13:32.763 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:32.763 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:13:32.764 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:13:32.764 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T16:13:32.764 INFO:teuthology.orchestra.run.smithi018.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:13:32.764 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:32.764 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:13:32.764 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:32.764 INFO:teuthology.orchestra.run.smithi018.stdout:Install 2 Packages 2024-03-25T16:13:32.764 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:32.764 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 332 k 2024-03-25T16:13:32.764 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 1.5 M 2024-03-25T16:13:32.764 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:13:32.899 INFO:teuthology.orchestra.run.smithi018.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 509 kB/s | 69 kB 00:00 2024-03-25T16:13:33.208 INFO:teuthology.orchestra.run.smithi018.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 594 kB/s | 264 kB 00:00 2024-03-25T16:13:33.209 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:33.209 INFO:teuthology.orchestra.run.smithi018.stdout:Total 745 kB/s | 332 kB 00:00 2024-03-25T16:13:33.210 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:13:33.219 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:13:33.219 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:13:33.256 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:13:33.256 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:13:33.412 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:13:33.589 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:13:33.618 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:13:33.634 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:13:33.634 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:33.635 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:13:33.635 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:34.409 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:13:34.765 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:13:34.765 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:34.765 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:13:34.765 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:34.765 INFO:teuthology.orchestra.run.smithi018.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:13:34.765 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:34.766 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:35.091 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install librados-devel 2024-03-25T16:13:35.671 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:50 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:35.826 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:35.827 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:35.827 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T16:13:35.827 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:35.827 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:13:35.827 INFO:teuthology.orchestra.run.smithi018.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:13:35.827 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:35.827 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:13:35.827 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:35.827 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T16:13:35.827 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:35.827 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 127 k 2024-03-25T16:13:35.827 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 456 k 2024-03-25T16:13:35.828 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:13:36.472 INFO:teuthology.orchestra.run.smithi018.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 196 kB/s | 127 kB 00:00 2024-03-25T16:13:36.473 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:36.473 INFO:teuthology.orchestra.run.smithi018.stdout:Total 196 kB/s | 127 kB 00:00 2024-03-25T16:13:36.473 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:13:36.480 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:13:36.480 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:13:36.539 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:13:36.539 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:13:36.692 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:13:36.807 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:37.250 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:37.540 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:37.540 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:37.540 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:13:37.540 INFO:teuthology.orchestra.run.smithi018.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:37.540 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:37.540 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:37.775 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install libcephfs2 2024-03-25T16:13:38.376 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:53 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:38.478 INFO:teuthology.orchestra.run.smithi018.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:38.534 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:38.535 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T16:13:38.535 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:38.604 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install libcephfs-devel 2024-03-25T16:13:39.190 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:54 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:39.344 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:39.345 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:39.345 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T16:13:39.345 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:39.345 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:13:39.346 INFO:teuthology.orchestra.run.smithi018.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:13:39.346 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:39.346 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:13:39.346 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:39.346 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T16:13:39.346 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:39.346 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 31 k 2024-03-25T16:13:39.346 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 140 k 2024-03-25T16:13:39.346 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:13:39.860 INFO:teuthology.orchestra.run.smithi018.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 61 kB/s | 31 kB 00:00 2024-03-25T16:13:39.860 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:39.861 INFO:teuthology.orchestra.run.smithi018.stdout:Total 60 kB/s | 31 kB 00:00 2024-03-25T16:13:39.861 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:13:39.865 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:13:39.865 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:13:39.891 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:13:39.891 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:13:40.028 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:13:40.127 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:40.500 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:40.762 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:40.762 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:40.763 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:13:40.763 INFO:teuthology.orchestra.run.smithi018.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:40.763 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:40.763 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:40.906 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install librados2 2024-03-25T16:13:41.484 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:56 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:41.583 INFO:teuthology.orchestra.run.smithi018.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:41.637 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:41.638 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T16:13:41.638 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:41.694 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install librbd1 2024-03-25T16:13:42.271 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:57 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:42.370 INFO:teuthology.orchestra.run.smithi018.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:42.425 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:42.426 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T16:13:42.426 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:42.477 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-rados 2024-03-25T16:13:43.051 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:58 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:43.151 INFO:teuthology.orchestra.run.smithi018.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:43.205 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:43.206 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T16:13:43.206 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:43.257 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-rgw 2024-03-25T16:13:43.831 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:58 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:43.930 INFO:teuthology.orchestra.run.smithi018.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:43.987 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:43.988 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T16:13:43.988 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:44.043 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-cephfs 2024-03-25T16:13:44.617 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:59 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:44.718 INFO:teuthology.orchestra.run.smithi018.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:44.773 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:44.774 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T16:13:44.774 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:44.827 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-rbd 2024-03-25T16:13:45.400 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:02:00 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:45.499 INFO:teuthology.orchestra.run.smithi018.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:45.554 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:45.555 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T16:13:45.555 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:45.609 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install rbd-fuse 2024-03-25T16:13:46.187 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:02:01 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:46.342 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:46.343 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:46.343 INFO:teuthology.orchestra.run.smithi018.stdout: Package Architecture Version Repository Size 2024-03-25T16:13:46.343 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:46.343 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:13:46.344 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:13:46.344 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:46.344 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:13:46.344 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:46.344 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T16:13:46.344 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:46.344 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 85 k 2024-03-25T16:13:46.344 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 231 k 2024-03-25T16:13:46.344 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:13:47.012 INFO:teuthology.orchestra.run.smithi018.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 128 kB/s | 85 kB 00:00 2024-03-25T16:13:47.013 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:47.013 INFO:teuthology.orchestra.run.smithi018.stdout:Total 127 kB/s | 85 kB 00:00 2024-03-25T16:13:47.013 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:13:47.021 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:13:47.021 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:13:47.081 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:13:47.081 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:13:47.222 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:13:47.322 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:47.714 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:48.001 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:48.002 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:48.002 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:13:48.002 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:48.002 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:48.002 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:48.231 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install rbd-mirror 2024-03-25T16:13:48.813 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:02:03 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:48.971 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:48.971 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:48.971 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T16:13:48.971 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:48.971 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:13:48.971 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:13:48.971 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:48.971 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:13:48.972 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:48.972 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T16:13:48.972 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:48.972 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 3.1 M 2024-03-25T16:13:48.972 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 13 M 2024-03-25T16:13:48.972 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:13:50.365 INFO:teuthology.orchestra.run.smithi018.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.2 MB/s | 3.1 MB 00:01 2024-03-25T16:13:50.366 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:50.366 INFO:teuthology.orchestra.run.smithi018.stdout:Total 2.2 MB/s | 3.1 MB 00:01 2024-03-25T16:13:50.367 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:13:50.378 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:13:50.379 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:13:50.458 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:13:50.459 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:13:50.913 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:13:50.967 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:50.990 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:50.990 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:50.990 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:13:50.990 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:13:50.990 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:13:50.990 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:52.154 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:52.155 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:52.155 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:13:52.155 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:52.155 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:52.155 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:52.402 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install rbd-nbd 2024-03-25T16:13:52.988 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:02:07 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:53.142 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:53.143 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:53.144 INFO:teuthology.orchestra.run.smithi018.stdout: Package Architecture Version Repository Size 2024-03-25T16:13:53.144 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:53.144 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:13:53.144 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:13:53.144 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:53.144 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:13:53.145 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:53.145 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T16:13:53.145 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:53.145 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 171 k 2024-03-25T16:13:53.145 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 494 k 2024-03-25T16:13:53.145 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:13:53.802 INFO:teuthology.orchestra.run.smithi018.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 261 kB/s | 171 kB 00:00 2024-03-25T16:13:53.803 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:53.803 INFO:teuthology.orchestra.run.smithi018.stdout:Total 260 kB/s | 171 kB 00:00 2024-03-25T16:13:53.803 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:13:53.812 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:13:53.812 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:13:53.872 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:13:53.873 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:13:54.023 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:13:54.138 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:54.538 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:54.821 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:54.821 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:54.821 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:13:54.821 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:54.822 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:54.822 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:13:55.050 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:13:55.625 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:02:10 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:13:55.781 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:13:55.782 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:55.782 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T16:13:55.782 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:55.783 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:13:55.783 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:13:55.783 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-25T16:13:55.783 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:13:55.783 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:55.783 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:13:55.783 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:13:55.783 INFO:teuthology.orchestra.run.smithi018.stdout:Install 2 Packages 2024-03-25T16:13:55.784 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:13:55.784 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 91 M 2024-03-25T16:13:55.784 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 498 M 2024-03-25T16:13:55.784 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:14:10.371 INFO:teuthology.orchestra.run.smithi018.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 1.5 MB/s | 22 MB 00:14 2024-03-25T16:14:45.908 INFO:teuthology.orchestra.run.smithi018.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 1.4 MB/s | 68 MB 00:50 2024-03-25T16:14:45.909 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:45.909 INFO:teuthology.orchestra.run.smithi018.stdout:Total 1.8 MB/s | 91 MB 00:50 2024-03-25T16:14:45.909 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:14:45.914 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:14:45.914 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:14:46.524 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:14:46.525 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:14:49.801 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:14:57.320 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:14:57.403 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:15:00.722 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:15:00.722 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:15:00.983 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:15:00.984 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:15:00.984 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:15:00.984 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:15:00.984 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:15:00.984 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:15:00.984 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:15:01.206 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:15:01.778 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:03:16 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:15:01.932 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:15:01.933 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:15:01.933 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T16:15:01.933 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:15:01.933 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:15:01.933 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:15:01.933 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:15:01.933 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:15:01.933 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:15:01.933 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T16:15:01.934 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:15:01.934 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 416 M 2024-03-25T16:15:01.934 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 1.7 G 2024-03-25T16:15:01.934 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:16:37.566 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 68 kB/s | 21 MB 05:10 2024-03-25T16:16:42.537 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-25T16:16:44.981 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:16:44.990 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:16:44.991 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T16:16:44.991 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:16:44.991 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:16:44.991 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:16:44.991 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrading: 2024-03-25T16:16:44.991 INFO:teuthology.orchestra.run.smithi169.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:16:44.991 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:16:44.991 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T16:16:44.991 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options 2024-03-25T16:16:44.991 INFO:teuthology.orchestra.run.smithi169.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:16:44.992 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:16:44.992 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:16:44.992 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:16:44.992 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:16:44.992 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:16:44.992 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:16:44.992 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:16:44.992 INFO:teuthology.orchestra.run.smithi169.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:16:44.992 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:16:44.992 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:16:44.993 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:16:44.993 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:16:44.993 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:16:44.993 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:16:44.993 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:16:44.993 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:16:44.993 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse 2024-03-25T16:16:44.993 INFO:teuthology.orchestra.run.smithi169.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:16:44.993 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:16:44.993 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:16:44.993 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:16:44.993 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:16:44.994 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:16:44.994 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:16:44.994 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:16:44.994 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:16:44.994 INFO:teuthology.orchestra.run.smithi169.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:16:44.994 INFO:teuthology.orchestra.run.smithi169.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:16:44.994 INFO:teuthology.orchestra.run.smithi169.stdout:Installing weak dependencies: 2024-03-25T16:16:44.994 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:16:44.994 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:16:44.994 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:16:44.994 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:16:44.994 INFO:teuthology.orchestra.run.smithi169.stdout:Install 30 Packages 2024-03-25T16:16:44.994 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrade 2 Packages 2024-03-25T16:16:44.994 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:16:44.995 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 58 M 2024-03-25T16:16:44.995 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:16:48.564 INFO:teuthology.orchestra.run.smithi169.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.7 MB/s | 5.3 MB 00:03 2024-03-25T16:16:48.655 INFO:teuthology.orchestra.run.smithi169.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 275 kB/s | 25 kB 00:00 2024-03-25T16:16:48.939 INFO:teuthology.orchestra.run.smithi169.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 2.6 MB/s | 742 kB 00:00 2024-03-25T16:16:49.172 INFO:teuthology.orchestra.run.smithi169.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 2.1 MB/s | 498 kB 00:00 2024-03-25T16:16:51.531 INFO:teuthology.orchestra.run.smithi169.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 2.2 MB/s | 5.1 MB 00:02 2024-03-25T16:16:51.631 INFO:teuthology.orchestra.run.smithi169.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 452 kB/s | 45 kB 00:00 2024-03-25T16:16:51.739 INFO:teuthology.orchestra.run.smithi169.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.2 MB/s | 131 kB 00:00 2024-03-25T16:16:51.915 INFO:teuthology.orchestra.run.smithi169.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 925 kB/s | 162 kB 00:00 2024-03-25T16:16:52.232 INFO:teuthology.orchestra.run.smithi169.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 1.0 MB/s | 320 kB 00:00 2024-03-25T16:16:52.490 INFO:teuthology.orchestra.run.smithi169.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.1 MB/s | 298 kB 00:00 2024-03-25T16:16:52.582 INFO:teuthology.orchestra.run.smithi169.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.1 MB/s | 99 kB 00:00 2024-03-25T16:16:52.849 INFO:teuthology.orchestra.run.smithi169.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 397 kB/s | 106 kB 00:00 2024-03-25T16:16:52.983 INFO:teuthology.orchestra.run.smithi169.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-03-25T16:16:53.033 INFO:teuthology.orchestra.run.smithi169.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-03-25T16:16:53.117 INFO:teuthology.orchestra.run.smithi169.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-03-25T16:16:53.159 INFO:teuthology.orchestra.run.smithi169.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:16:53.192 INFO:teuthology.orchestra.run.smithi169.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:16:53.718 INFO:teuthology.orchestra.run.smithi169.stdout:(18/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 1.2 MB/s | 9.7 MB 00:08 2024-03-25T16:16:54.168 INFO:teuthology.orchestra.run.smithi169.stdout:(19/32): qatlib-23.11.0-1.el9.x86_64.rpm 222 kB/s | 216 kB 00:00 2024-03-25T16:16:54.318 INFO:teuthology.orchestra.run.smithi169.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 58 kB/s | 35 kB 00:00 2024-03-25T16:16:54.726 INFO:teuthology.orchestra.run.smithi169.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 757 kB/s | 309 kB 00:00 2024-03-25T16:16:54.968 INFO:teuthology.orchestra.run.smithi169.stdout:(22/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 56 kB/s | 44 kB 00:00 2024-03-25T16:16:55.127 INFO:teuthology.orchestra.run.smithi169.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 164 kB/s | 26 kB 00:00 2024-03-25T16:16:55.286 INFO:teuthology.orchestra.run.smithi169.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 307 kB/s | 49 kB 00:00 2024-03-25T16:16:55.444 INFO:teuthology.orchestra.run.smithi169.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 425 kB/s | 67 kB 00:00 2024-03-25T16:16:55.644 INFO:teuthology.orchestra.run.smithi169.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2024-03-25T16:16:55.695 INFO:teuthology.orchestra.run.smithi169.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.8 MB/s | 192 kB 00:00 2024-03-25T16:16:55.795 INFO:teuthology.orchestra.run.smithi169.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-25T16:16:57.396 INFO:teuthology.orchestra.run.smithi169.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 1.7 MB/s | 4.4 MB 00:02 2024-03-25T16:16:57.871 INFO:teuthology.orchestra.run.smithi169.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 1.6 MB/s | 3.4 MB 00:02 2024-03-25T16:16:59.512 INFO:teuthology.orchestra.run.smithi169.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 1.5 MB/s | 3.1 MB 00:02 2024-03-25T16:17:01.696 INFO:teuthology.orchestra.run.smithi169.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 20 MB 00:16 2024-03-25T16:17:01.699 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:01.699 INFO:teuthology.orchestra.run.smithi169.stdout:Total 3.5 MB/s | 58 MB 00:16 2024-03-25T16:17:01.732 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:17:01.770 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:17:01.770 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:17:02.258 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:17:02.259 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:17:03.713 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:17:03.804 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:17:03.834 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:17:03.859 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:17:03.890 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:17:03.906 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:17:04.104 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:17:04.166 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:17:04.481 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:17:04.552 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:17:04.653 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:17:04.739 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:17:04.771 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:17:04.825 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:17:04.853 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:17:04.897 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:17:04.965 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:17:04.989 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:17:05.052 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:17:05.079 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:17:05.188 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:17:05.223 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:17:05.263 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:17:05.296 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:17:05.325 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:17:05.358 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:17:05.680 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:17:05.704 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:17:05.786 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:17:05.818 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:17:05.866 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:17:06.319 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:17:06.349 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:17:06.420 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:17:06.488 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:17:06.517 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:17:08.686 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:17:08.731 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:17:09.282 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:17:09.318 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:17:09.418 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:17:09.418 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:17:09.419 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:17:09.419 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:17:09.453 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:17:18.743 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:17:18.744 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /sys 2024-03-25T16:17:18.744 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /proc 2024-03-25T16:17:18.744 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /mnt 2024-03-25T16:17:18.744 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /var/tmp 2024-03-25T16:17:18.744 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /home 2024-03-25T16:17:18.745 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /root 2024-03-25T16:17:18.745 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /tmp 2024-03-25T16:17:18.745 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:17:19.678 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:17:19.703 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:17:19.703 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:17:19.703 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:17:19.703 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:17:19.703 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:17:19.703 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:17:19.705 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:17:19.789 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:17:19.790 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:17:20.944 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:17:20.945 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:17:20.945 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:17:20.945 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:17:20.946 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:17:20.946 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:17:20.946 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:17:20.946 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:17:20.946 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:17:20.946 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:17:20.947 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:17:20.947 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:17:20.947 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:17:20.947 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:17:20.947 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:17:20.947 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:17:20.948 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:17:20.948 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:17:20.948 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:17:20.948 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:17:20.948 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:17:20.948 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:17:20.949 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:17:20.949 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:17:20.949 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:17:20.949 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:17:20.949 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:17:20.949 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:17:20.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:17:20.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:17:20.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:17:20.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:17:20.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:17:20.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:17:22.122 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:17:22.123 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:17:22.123 INFO:teuthology.orchestra.run.smithi169.stdout:Upgraded: 2024-03-25T16:17:22.123 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:22.123 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:22.123 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:17:22.123 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:17:22.123 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:22.123 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:22.123 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:22.123 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:22.123 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:17:22.123 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:17:22.124 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:17:22.124 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:22.124 INFO:teuthology.orchestra.run.smithi169.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:17:22.124 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:17:22.124 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:17:22.124 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:22.124 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:17:22.124 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:22.124 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:17:22.124 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:17:22.124 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:17:22.124 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:22.124 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:22.124 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:22.125 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:17:22.125 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:22.125 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:22.125 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:22.125 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:17:22.125 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:17:22.125 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:17:22.125 INFO:teuthology.orchestra.run.smithi169.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:17:22.125 INFO:teuthology.orchestra.run.smithi169.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:17:22.125 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:17:22.125 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:17:22.490 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-test 2024-03-25T16:17:23.057 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:41 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:17:23.211 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:17:23.212 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:17:23.212 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T16:17:23.212 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:17:23.212 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:17:23.212 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:17:23.212 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T16:17:23.212 INFO:teuthology.orchestra.run.smithi169.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:17:23.212 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:17:23.212 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:17:23.212 INFO:teuthology.orchestra.run.smithi169.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:17:23.213 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:17:23.213 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:17:23.213 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:17:23.213 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:17:23.213 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2024-03-25T16:17:23.213 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:17:23.213 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 48 M 2024-03-25T16:17:23.214 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 204 M 2024-03-25T16:17:23.214 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:17:23.427 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-25T16:17:23.478 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T16:17:23.519 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-25T16:17:23.645 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T16:17:24.062 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 208 kB/s | 163 kB 00:00 2024-03-25T16:17:40.702 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 2.7 MB/s | 47 MB 00:17 2024-03-25T16:17:40.704 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:40.704 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.8 MB/s | 48 MB 00:17 2024-03-25T16:17:40.718 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:17:40.738 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:17:40.738 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:17:41.116 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:17:41.116 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:17:41.699 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:17:41.777 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:17:41.823 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:17:41.863 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:17:41.907 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:17:41.929 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:17:46.520 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:17:46.596 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:17:48.949 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:17:48.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:17:48.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:17:48.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:17:48.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:17:48.950 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:17:49.340 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:17:49.340 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:17:49.340 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:17:49.340 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:49.340 INFO:teuthology.orchestra.run.smithi169.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:17:49.340 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:17:49.340 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:17:49.340 INFO:teuthology.orchestra.run.smithi169.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:17:49.341 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:17:49.341 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:17:49.341 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:17:49.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph 2024-03-25T16:17:50.245 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:08 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:17:50.402 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:17:50.405 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================= 2024-03-25T16:17:50.405 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T16:17:50.405 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================= 2024-03-25T16:17:50.405 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:17:50.406 INFO:teuthology.orchestra.run.smithi169.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:17:50.406 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T16:17:50.406 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:17:50.406 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:17:50.406 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:17:50.406 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:17:50.406 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:17:50.406 INFO:teuthology.orchestra.run.smithi169.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:17:50.407 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:17:50.407 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:17:50.407 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:17:50.407 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:17:50.407 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:17:50.407 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:17:50.407 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:17:50.407 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:17:50.407 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:17:50.407 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:17:50.407 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:17:50.408 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:17:50.408 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:17:50.408 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:17:50.408 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:17:50.408 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:17:50.408 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:17:50.408 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:17:50.408 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:17:50.408 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:17:50.408 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:17:50.408 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:17:50.409 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:17:50.409 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:17:50.409 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:17:50.409 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:17:50.409 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:17:50.409 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:17:50.409 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:17:50.409 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:17:50.409 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:17:50.409 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================= 2024-03-25T16:17:50.409 INFO:teuthology.orchestra.run.smithi169.stdout:Install 36 Packages 2024-03-25T16:17:50.410 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:17:50.411 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 30 M 2024-03-25T16:17:50.411 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 107 M 2024-03-25T16:17:50.411 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:17:51.246 INFO:teuthology.orchestra.run.smithi169.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 19 kB/s | 6.6 kB 00:00 2024-03-25T16:17:53.071 INFO:teuthology.orchestra.run.smithi169.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.0 MB/s | 2.1 MB 00:02 2024-03-25T16:17:53.530 INFO:teuthology.orchestra.run.smithi169.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 581 kB/s | 1.5 MB 00:02 2024-03-25T16:17:53.763 INFO:teuthology.orchestra.run.smithi169.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.0 MB/s | 247 kB 00:00 2024-03-25T16:17:53.980 INFO:teuthology.orchestra.run.smithi169.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 862 kB/s | 187 kB 00:00 2024-03-25T16:17:54.039 INFO:teuthology.orchestra.run.smithi169.stdout:(6/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 1.7 MB/s | 4.7 MB 00:02 2024-03-25T16:17:54.064 INFO:teuthology.orchestra.run.smithi169.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2024-03-25T16:17:54.089 INFO:teuthology.orchestra.run.smithi169.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 634 kB/s | 32 kB 00:00 2024-03-25T16:17:54.114 INFO:teuthology.orchestra.run.smithi169.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 943 kB/s | 47 kB 00:00 2024-03-25T16:17:54.148 INFO:teuthology.orchestra.run.smithi169.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 660 kB/s | 39 kB 00:00 2024-03-25T16:17:54.215 INFO:teuthology.orchestra.run.smithi169.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.4 MB/s | 241 kB 00:00 2024-03-25T16:17:54.373 INFO:teuthology.orchestra.run.smithi169.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 5.1 MB/s | 1.1 MB 00:00 2024-03-25T16:17:54.399 INFO:teuthology.orchestra.run.smithi169.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 561 kB/s | 103 kB 00:00 2024-03-25T16:17:54.432 INFO:teuthology.orchestra.run.smithi169.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2024-03-25T16:17:54.466 INFO:teuthology.orchestra.run.smithi169.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.7 MB/s | 187 kB 00:00 2024-03-25T16:17:54.641 INFO:teuthology.orchestra.run.smithi169.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 598 kB/s | 125 kB 00:00 2024-03-25T16:17:55.133 INFO:teuthology.orchestra.run.smithi169.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 33 kB/s | 22 kB 00:00 2024-03-25T16:17:55.159 INFO:teuthology.orchestra.run.smithi169.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 292 kB/s | 151 kB 00:00 2024-03-25T16:17:55.476 INFO:teuthology.orchestra.run.smithi169.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 127 kB/s | 43 kB 00:00 2024-03-25T16:17:55.584 INFO:teuthology.orchestra.run.smithi169.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 404 kB/s | 172 kB 00:00 2024-03-25T16:17:55.768 INFO:teuthology.orchestra.run.smithi169.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 58 kB/s | 11 kB 00:00 2024-03-25T16:17:56.026 INFO:teuthology.orchestra.run.smithi169.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 68 kB/s | 18 kB 00:00 2024-03-25T16:17:56.068 INFO:teuthology.orchestra.run.smithi169.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 605 kB/s | 358 kB 00:00 2024-03-25T16:17:56.293 INFO:teuthology.orchestra.run.smithi169.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 87 kB/s | 23 kB 00:00 2024-03-25T16:17:56.335 INFO:teuthology.orchestra.run.smithi169.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 73 kB/s | 19 kB 00:00 2024-03-25T16:17:56.553 INFO:teuthology.orchestra.run.smithi169.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 76 kB/s | 20 kB 00:00 2024-03-25T16:17:56.678 INFO:teuthology.orchestra.run.smithi169.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 135 kB/s | 46 kB 00:00 2024-03-25T16:17:56.903 INFO:teuthology.orchestra.run.smithi169.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 225 kB/s | 79 kB 00:00 2024-03-25T16:17:57.170 INFO:teuthology.orchestra.run.smithi169.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 62 kB/s | 16 kB 00:00 2024-03-25T16:17:57.195 INFO:teuthology.orchestra.run.smithi169.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 527 kB/s | 272 kB 00:00 2024-03-25T16:17:57.462 INFO:teuthology.orchestra.run.smithi169.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 134 kB/s | 36 kB 00:00 2024-03-25T16:17:57.596 INFO:teuthology.orchestra.run.smithi169.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 211 kB/s | 90 kB 00:00 2024-03-25T16:17:57.971 INFO:teuthology.orchestra.run.smithi169.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 452 kB/s | 230 kB 00:00 2024-03-25T16:17:58.188 INFO:teuthology.orchestra.run.smithi169.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 722 kB/s | 427 kB 00:00 2024-03-25T16:17:58.238 INFO:teuthology.orchestra.run.smithi169.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 75 kB/s | 20 kB 00:00 2024-03-25T16:18:01.721 INFO:teuthology.orchestra.run.smithi169.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.9 MB/s | 17 MB 00:08 2024-03-25T16:18:01.725 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:01.725 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.6 MB/s | 30 MB 00:11 2024-03-25T16:18:01.750 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:18:01.793 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:18:01.793 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:18:02.198 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:18:02.198 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:18:03.642 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:18:03.715 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:18:03.749 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:18:03.776 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:18:03.846 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:18:03.891 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:18:03.925 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:18:04.015 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:18:04.093 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:18:04.135 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:18:04.207 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:18:04.261 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:18:04.300 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:18:04.348 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:18:04.404 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:18:04.588 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:18:04.646 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:18:04.684 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:18:04.718 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:18:04.749 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:18:04.825 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:18:04.859 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:18:04.893 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:18:04.933 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:18:04.968 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:18:05.002 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:18:05.092 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:18:05.133 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:18:05.209 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:18:05.341 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:18:05.459 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:18:05.619 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:18:05.639 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:18:05.665 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:18:05.665 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:18:05.665 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:18:05.665 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:18:05.665 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:18:05.665 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:07.098 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:18:07.124 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:18:07.124 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:18:07.124 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:18:07.124 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:18:07.124 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:18:07.124 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:07.565 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:18:07.589 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:18:07.589 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:18:07.589 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:18:07.589 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:18:07.589 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:18:07.589 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:07.799 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:18:07.822 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:18:07.823 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:18:07.823 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:18:07.823 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:18:07.823 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:18:07.823 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:07.887 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:18:09.598 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:18:09.598 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:18:09.598 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:18:09.598 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:18:09.598 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:18:09.598 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:18:09.599 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:18:09.599 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:18:09.599 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:18:09.599 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:18:09.599 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:18:09.599 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:18:09.599 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:18:09.599 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:18:09.599 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:18:09.599 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:18:09.599 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:18:09.599 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:18:09.599 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:18:09.600 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:18:09.600 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:18:09.600 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:18:09.600 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:18:09.600 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:18:09.600 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:18:09.600 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:18:09.600 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:18:09.600 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:18:09.600 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:18:09.601 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:18:09.601 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:18:09.601 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:18:09.601 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:18:09.601 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:18:09.601 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:18:09.602 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:18:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:18:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:10.600 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:18:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:10.600 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:18:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:18:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:18:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:18:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:18:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:18:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:18:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:18:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:18:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:18:10.601 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:18:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:18:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:18:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:18:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:18:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:18:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:18:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:18:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:18:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:18:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:18:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:18:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:18:10.602 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:18:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:18:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:18:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:18:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:18:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:18:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:18:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:18:10.603 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:10.603 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:18:10.979 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-base 2024-03-25T16:18:11.547 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:29 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:18:11.644 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:18:11.695 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:18:11.696 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T16:18:11.696 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:18:11.764 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install cephadm 2024-03-25T16:18:12.322 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:30 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:18:12.470 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:18:12.471 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:18:12.471 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T16:18:12.471 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:18:12.471 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:18:12.471 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:18:12.471 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:12.472 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:18:12.472 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:18:12.472 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T16:18:12.472 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:12.472 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 747 k 2024-03-25T16:18:12.472 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 754 k 2024-03-25T16:18:12.472 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:18:13.608 INFO:teuthology.orchestra.run.smithi169.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 658 kB/s | 747 kB 00:01 2024-03-25T16:18:13.608 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:13.609 INFO:teuthology.orchestra.run.smithi169.stdout:Total 657 kB/s | 747 kB 00:01 2024-03-25T16:18:13.609 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:18:13.615 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:18:13.615 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:18:13.620 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:18:13.621 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:18:13.698 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:18:14.200 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:18:14.295 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:18:14.709 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:18:14.983 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:18:14.983 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:14.983 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:18:14.984 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:18:14.984 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:14.984 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:18:15.213 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:18:15.791 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:33 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:18:15.941 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:18:15.941 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:18:15.942 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T16:18:15.942 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:18:15.942 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:18:15.942 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:18:15.942 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:15.942 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:18:15.942 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:18:15.942 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T16:18:15.942 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:15.942 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 145 k 2024-03-25T16:18:15.942 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 436 k 2024-03-25T16:18:15.942 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:18:16.547 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 240 kB/s | 145 kB 00:00 2024-03-25T16:18:16.547 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:16.548 INFO:teuthology.orchestra.run.smithi169.stdout:Total 240 kB/s | 145 kB 00:00 2024-03-25T16:18:16.548 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:18:16.556 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:18:16.556 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:18:16.610 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:18:16.610 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:18:16.754 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:18:16.806 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:18:16.830 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:18:16.831 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:18:16.831 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:18:16.831 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:17.836 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:18:17.836 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:17.836 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:18:17.836 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:17.837 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:17.837 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:18:18.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr 2024-03-25T16:18:18.696 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:36 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:18:18.795 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:18:18.847 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:18:18.848 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T16:18:18.848 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:18:18.912 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:18:19.475 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:37 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:18:19.627 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:18:19.631 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-03-25T16:18:19.631 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T16:18:19.631 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-03-25T16:18:19.631 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:18:19.631 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:18:19.631 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T16:18:19.631 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:18:19.632 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:18:19.632 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:18:19.632 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:18:19.632 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:18:19.632 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:18:19.632 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:18:19.632 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:18:19.632 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:18:19.632 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:18:19.632 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:18:19.632 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:19.632 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:18:19.632 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-03-25T16:18:19.632 INFO:teuthology.orchestra.run.smithi169.stdout:Install 12 Packages 2024-03-25T16:18:19.632 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:19.633 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 6.9 M 2024-03-25T16:18:19.633 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 26 M 2024-03-25T16:18:19.633 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:18:20.202 INFO:teuthology.orchestra.run.smithi169.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 35 kB/s | 15 kB 00:00 2024-03-25T16:18:20.269 INFO:teuthology.orchestra.run.smithi169.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 62 kB/s | 31 kB 00:00 2024-03-25T16:18:20.427 INFO:teuthology.orchestra.run.smithi169.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-03-25T16:18:20.453 INFO:teuthology.orchestra.run.smithi169.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 238 kB 00:00 2024-03-25T16:18:20.653 INFO:teuthology.orchestra.run.smithi169.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.7 MB/s | 863 kB 00:00 2024-03-25T16:18:20.845 INFO:teuthology.orchestra.run.smithi169.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 101 kB/s | 19 kB 00:00 2024-03-25T16:18:20.870 INFO:teuthology.orchestra.run.smithi169.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.3 MB/s | 548 kB 00:00 2024-03-25T16:18:20.954 INFO:teuthology.orchestra.run.smithi169.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.7 MB/s | 144 kB 00:00 2024-03-25T16:18:21.030 INFO:teuthology.orchestra.run.smithi169.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 409 kB/s | 31 kB 00:00 2024-03-25T16:18:21.113 INFO:teuthology.orchestra.run.smithi169.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-25T16:18:21.389 INFO:teuthology.orchestra.run.smithi169.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.8 MB/s | 2.0 MB 00:00 2024-03-25T16:18:22.188 INFO:teuthology.orchestra.run.smithi169.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 752 kB/s | 1.8 MB 00:02 2024-03-25T16:18:22.192 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:22.192 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.7 MB/s | 6.9 MB 00:02 2024-03-25T16:18:22.212 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:18:22.231 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:18:22.231 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:18:22.408 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:18:22.408 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:18:22.945 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:18:23.022 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:18:23.065 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:18:23.112 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:18:23.151 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:18:23.258 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:18:23.394 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:18:23.479 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:18:23.522 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:18:23.559 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:18:23.591 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:18:23.782 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 2.1 MB/s | 416 MB 03:21 2024-03-25T16:18:23.783 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:23.783 INFO:teuthology.orchestra.run.smithi018.stdout:Total 2.1 MB/s | 416 MB 03:21 2024-03-25T16:18:23.784 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:18:23.787 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:18:23.787 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:18:23.835 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:18:23.866 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:18:24.731 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:18:24.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:18:24.731 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:18:24.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:18:24.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:18:24.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:18:24.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:18:24.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:18:24.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:18:24.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:18:24.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:18:24.732 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:18:25.091 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:18:25.091 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:25.091 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:18:25.091 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:18:25.091 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:18:25.091 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:18:25.091 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:18:25.092 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:18:25.092 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:18:25.092 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:18:25.092 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:18:25.092 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:18:25.092 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:18:25.092 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:18:25.092 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:18:25.093 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:25.093 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:18:25.377 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:18:25.936 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:43 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:18:26.087 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:18:26.088 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-25T16:18:26.088 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T16:18:26.088 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-25T16:18:26.088 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:18:26.088 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:18:26.088 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T16:18:26.089 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:18:26.089 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:18:26.089 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:18:26.089 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:18:26.089 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:18:26.089 INFO:teuthology.orchestra.run.smithi169.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:18:26.089 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:18:26.089 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:18:26.089 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:18:26.089 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:18:26.089 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:18:26.089 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:26.089 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:18:26.090 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-03-25T16:18:26.090 INFO:teuthology.orchestra.run.smithi169.stdout:Install 12 Packages 2024-03-25T16:18:26.090 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:26.090 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 37 M 2024-03-25T16:18:26.090 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 209 M 2024-03-25T16:18:26.091 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:18:26.376 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:18:26.376 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:18:26.428 INFO:teuthology.orchestra.run.smithi169.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 289 kB/s | 32 kB 00:00 2024-03-25T16:18:26.495 INFO:teuthology.orchestra.run.smithi169.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 392 kB/s | 17 kB 00:00 2024-03-25T16:18:26.529 INFO:teuthology.orchestra.run.smithi169.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T16:18:26.729 INFO:teuthology.orchestra.run.smithi169.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2024-03-25T16:18:26.879 INFO:teuthology.orchestra.run.smithi169.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-25T16:18:26.938 INFO:teuthology.orchestra.run.smithi169.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.8 MB/s | 3.0 MB 00:00 2024-03-25T16:18:26.963 INFO:teuthology.orchestra.run.smithi169.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-25T16:18:27.464 INFO:teuthology.orchestra.run.smithi169.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-03-25T16:18:28.355 INFO:teuthology.orchestra.run.smithi169.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 219 kB/s | 195 kB 00:00 2024-03-25T16:18:28.606 INFO:teuthology.orchestra.run.smithi169.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 982 kB/s | 245 kB 00:00 2024-03-25T16:18:28.831 INFO:teuthology.orchestra.run.smithi169.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 430 kB/s | 803 kB 00:01 2024-03-25T16:18:34.672 INFO:teuthology.orchestra.run.smithi169.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 906 kB/s | 7.4 MB 00:08 2024-03-25T16:18:34.676 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:34.676 INFO:teuthology.orchestra.run.smithi169.stdout:Total 4.3 MB/s | 37 MB 00:08 2024-03-25T16:18:34.720 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:18:34.745 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:18:34.746 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:18:35.110 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:18:35.110 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:18:35.809 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:18:35.887 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:18:35.926 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:18:36.024 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:18:36.062 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:18:36.241 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:18:36.270 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:18:36.352 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:18:37.005 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:18:37.116 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:18:38.353 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:18:39.389 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:18:39.420 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:18:41.886 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:18:41.887 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:18:41.887 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:18:41.887 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:18:41.887 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:18:41.887 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:18:41.887 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:18:41.887 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:18:41.887 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:18:41.887 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:18:41.887 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:18:41.887 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:18:42.302 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:18:42.302 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:42.302 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:18:42.302 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:18:42.302 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:18:42.302 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:18:42.302 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:18:42.302 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:18:42.302 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:18:42.302 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:18:42.302 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:18:42.303 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:18:42.303 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:18:42.303 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:18:42.303 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:18:42.303 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:42.303 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:18:42.700 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:18:43.275 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:01 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:18:43.426 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:18:43.427 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T16:18:43.427 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T16:18:43.427 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T16:18:43.427 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:18:43.427 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:18:43.427 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T16:18:43.427 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:18:43.427 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:18:43.428 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:18:43.428 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:18:43.428 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:18:43.428 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:18:43.428 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:18:43.428 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:18:43.428 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:18:43.428 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:18:43.428 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:18:43.428 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:18:43.428 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:43.428 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:18:43.428 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T16:18:43.428 INFO:teuthology.orchestra.run.smithi169.stdout:Install 13 Packages 2024-03-25T16:18:43.428 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:43.429 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 2.0 M 2024-03-25T16:18:43.429 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 26 M 2024-03-25T16:18:43.429 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:18:43.569 INFO:teuthology.orchestra.run.smithi169.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 253 kB/s | 24 kB 00:00 2024-03-25T16:18:43.612 INFO:teuthology.orchestra.run.smithi169.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 131 kB/s | 18 kB 00:00 2024-03-25T16:18:43.662 INFO:teuthology.orchestra.run.smithi169.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-25T16:18:43.687 INFO:teuthology.orchestra.run.smithi169.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:18:43.712 INFO:teuthology.orchestra.run.smithi169.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:18:43.738 INFO:teuthology.orchestra.run.smithi169.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 871 kB/s | 43 kB 00:00 2024-03-25T16:18:43.971 INFO:teuthology.orchestra.run.smithi169.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 124 kB/s | 32 kB 00:00 2024-03-25T16:18:43.996 INFO:teuthology.orchestra.run.smithi169.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-03-25T16:18:44.030 INFO:teuthology.orchestra.run.smithi169.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 94 kB/s | 52 kB 00:00 2024-03-25T16:18:44.238 INFO:teuthology.orchestra.run.smithi169.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 758 kB/s | 203 kB 00:00 2024-03-25T16:18:44.314 INFO:teuthology.orchestra.run.smithi169.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-03-25T16:18:44.381 INFO:teuthology.orchestra.run.smithi169.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 2.7 MB/s | 1.0 MB 00:00 2024-03-25T16:18:44.422 INFO:teuthology.orchestra.run.smithi169.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 151 kB/s | 59 kB 00:00 2024-03-25T16:18:44.427 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:44.427 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-25T16:18:44.427 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:18:44.441 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:18:44.441 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:18:44.535 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:18:44.535 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:18:45.047 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:18:45.211 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:18:45.273 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:18:45.330 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:18:45.366 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:18:45.407 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:18:45.501 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:18:45.602 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:18:45.651 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:18:46.379 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:18:46.431 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:18:46.465 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:18:46.513 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:18:46.534 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:18:47.340 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:18:47.340 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:18:47.340 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:18:47.340 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:18:47.340 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:18:47.341 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:18:47.341 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:18:47.341 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:18:47.341 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:18:47.341 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:18:47.341 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:18:47.341 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:18:47.341 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:18:47.741 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:18:47.741 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:47.741 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:18:47.741 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:18:47.741 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:18:47.741 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:18:47.741 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:18:47.741 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:18:47.741 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:18:47.741 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:18:47.742 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:18:47.742 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:18:47.742 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:18:47.742 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:18:47.742 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:18:47.742 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:18:47.742 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:47.742 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:18:48.010 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:18:48.579 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:06 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:18:48.729 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:18:48.730 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T16:18:48.730 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T16:18:48.730 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T16:18:48.730 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:18:48.730 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:18:48.730 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T16:18:48.730 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:18:48.731 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:18:48.731 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:18:48.731 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:18:48.731 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:18:48.731 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:48.731 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:18:48.731 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T16:18:48.731 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2024-03-25T16:18:48.731 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:48.732 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 6.8 M 2024-03-25T16:18:48.732 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 33 M 2024-03-25T16:18:48.732 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:18:49.533 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 947 kB/s | 248 kB 00:00 2024-03-25T16:18:49.833 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 5.8 MB 00:00 2024-03-25T16:18:49.992 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-25T16:18:50.017 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 314 kB/s | 58 kB 00:00 2024-03-25T16:18:50.059 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 764 kB/s | 51 kB 00:00 2024-03-25T16:18:50.209 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 147 kB/s | 138 kB 00:00 2024-03-25T16:18:50.211 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:50.211 INFO:teuthology.orchestra.run.smithi169.stdout:Total 4.6 MB/s | 6.8 MB 00:01 2024-03-25T16:18:50.224 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:18:50.234 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:18:50.234 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:18:50.320 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:18:50.321 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:18:50.614 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:18:50.782 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:18:50.846 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:18:51.358 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:18:51.431 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:18:51.492 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:18:51.516 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:18:52.344 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:18:52.344 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:18:52.344 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:18:52.345 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:18:52.345 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:18:52.345 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:18:52.635 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:18:52.635 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:52.635 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:18:52.635 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:18:52.635 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:18:52.635 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:18:52.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:18:52.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:18:52.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:18:52.636 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:52.636 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:18:52.907 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-fuse 2024-03-25T16:18:53.475 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:11 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:18:53.626 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:18:53.627 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:18:53.627 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T16:18:53.627 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:18:53.627 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:18:53.627 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:18:53.627 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T16:18:53.627 INFO:teuthology.orchestra.run.smithi169.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:18:53.627 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:53.627 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:18:53.627 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:18:53.627 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2024-03-25T16:18:53.628 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:53.628 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 964 k 2024-03-25T16:18:53.628 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 2.8 M 2024-03-25T16:18:53.628 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:18:53.862 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 336 kB/s | 79 kB 00:00 2024-03-25T16:18:55.729 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 421 kB/s | 885 kB 00:02 2024-03-25T16:18:55.730 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:55.731 INFO:teuthology.orchestra.run.smithi169.stdout:Total 458 kB/s | 964 kB 00:02 2024-03-25T16:18:55.731 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:18:55.754 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:18:55.755 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:18:55.830 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:18:55.830 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:18:56.020 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:18:56.155 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:18:56.221 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:18:56.964 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:18:56.964 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:18:57.255 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:18:57.255 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:57.255 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:18:57.255 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:18:57.255 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:57.255 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:18:57.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-volume 2024-03-25T16:18:58.129 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:16 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:18:58.286 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:18:58.287 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:18:58.287 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-03-25T16:18:58.287 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:18:58.287 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:18:58.287 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:18:58.287 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T16:18:58.287 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:18:58.288 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:58.288 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:18:58.288 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:18:58.288 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2024-03-25T16:18:58.288 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:18:58.288 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 332 k 2024-03-25T16:18:58.288 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 1.5 M 2024-03-25T16:18:58.289 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:18:58.392 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 669 kB/s | 69 kB 00:00 2024-03-25T16:18:59.342 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 250 kB/s | 264 kB 00:01 2024-03-25T16:18:59.344 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:59.344 INFO:teuthology.orchestra.run.smithi169.stdout:Total 315 kB/s | 332 kB 00:01 2024-03-25T16:18:59.344 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:18:59.354 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:18:59.354 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:18:59.397 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:18:59.398 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:18:59.554 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:18:59.743 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:18:59.786 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:18:59.802 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:18:59.802 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:18:59.802 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:18:59.802 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:00.556 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:19:00.910 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:19:00.911 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:00.911 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:19:00.911 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:19:00.911 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:19:00.911 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:00.911 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:19:01.249 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados-devel 2024-03-25T16:19:01.831 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:19 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:19:01.982 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:19:01.983 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:01.983 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T16:19:01.983 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:01.983 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:19:01.983 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:19:01.983 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:01.983 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:19:01.983 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:01.984 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T16:19:01.984 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:01.984 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 127 k 2024-03-25T16:19:01.984 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 456 k 2024-03-25T16:19:01.984 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:19:02.648 INFO:teuthology.orchestra.run.smithi169.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 191 kB/s | 127 kB 00:00 2024-03-25T16:19:02.649 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:02.649 INFO:teuthology.orchestra.run.smithi169.stdout:Total 190 kB/s | 127 kB 00:00 2024-03-25T16:19:02.649 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:19:02.654 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:19:02.655 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:19:02.712 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:19:02.713 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:19:02.861 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:19:02.963 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:03.449 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:03.735 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:03.736 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:03.736 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:19:03.736 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:03.736 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:03.736 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:19:03.991 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs2 2024-03-25T16:19:04.573 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:22 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:19:04.675 INFO:teuthology.orchestra.run.smithi169.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:19:04.728 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:19:04.729 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T16:19:04.729 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:19:04.802 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs-devel 2024-03-25T16:19:05.373 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:23 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:19:05.524 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:19:05.524 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:05.524 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T16:19:05.525 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:05.525 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:19:05.525 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:19:05.525 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:05.525 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:19:05.525 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:05.525 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T16:19:05.525 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:05.525 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 31 k 2024-03-25T16:19:05.525 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 140 k 2024-03-25T16:19:05.525 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:19:06.059 INFO:teuthology.orchestra.run.smithi169.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 58 kB/s | 31 kB 00:00 2024-03-25T16:19:06.059 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:06.060 INFO:teuthology.orchestra.run.smithi169.stdout:Total 58 kB/s | 31 kB 00:00 2024-03-25T16:19:06.060 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:19:06.064 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:19:06.064 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:19:06.094 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:19:06.094 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:19:06.225 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:19:06.350 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:06.729 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:06.998 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:06.999 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:06.999 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:19:06.999 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:06.999 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:06.999 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:19:07.156 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados2 2024-03-25T16:19:07.727 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:25 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:19:07.826 INFO:teuthology.orchestra.run.smithi169.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:19:07.878 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:19:07.879 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T16:19:07.879 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:19:07.932 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librbd1 2024-03-25T16:19:08.501 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:26 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:19:08.599 INFO:teuthology.orchestra.run.smithi169.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:19:08.650 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:19:08.651 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T16:19:08.651 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:19:08.709 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rados 2024-03-25T16:19:09.274 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:27 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:19:09.370 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:19:09.422 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:19:09.423 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T16:19:09.423 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:19:09.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rgw 2024-03-25T16:19:10.046 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:28 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:19:10.142 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:19:10.194 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:19:10.195 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T16:19:10.195 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:19:10.251 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-cephfs 2024-03-25T16:19:10.820 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:28 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:19:10.916 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:19:10.968 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:19:10.969 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T16:19:10.969 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:19:11.010 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:19:11.019 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rbd 2024-03-25T16:19:11.104 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:19:11.584 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:29 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:19:11.681 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:19:11.732 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:19:11.733 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T16:19:11.733 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:19:11.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-fuse 2024-03-25T16:19:12.349 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:30 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:19:12.498 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:19:12.499 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:12.499 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-03-25T16:19:12.499 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:12.499 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:19:12.499 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:19:12.499 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:12.499 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:19:12.499 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:12.499 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T16:19:12.499 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:12.499 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 85 k 2024-03-25T16:19:12.500 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 231 k 2024-03-25T16:19:12.500 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:19:13.263 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 112 kB/s | 85 kB 00:00 2024-03-25T16:19:13.264 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:13.264 INFO:teuthology.orchestra.run.smithi169.stdout:Total 112 kB/s | 85 kB 00:00 2024-03-25T16:19:13.264 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:19:13.271 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:19:13.272 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:19:13.330 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:19:13.331 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:19:13.470 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:19:13.578 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:13.962 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:14.250 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:14.250 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:14.250 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:19:14.250 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:14.250 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:14.250 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:19:14.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-mirror 2024-03-25T16:19:15.048 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:33 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:19:15.202 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:19:15.202 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:15.202 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T16:19:15.202 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:15.202 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:19:15.203 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:19:15.203 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:15.203 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:19:15.203 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:15.203 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T16:19:15.203 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:15.203 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 3.1 M 2024-03-25T16:19:15.203 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 13 M 2024-03-25T16:19:15.203 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:19:16.069 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:19:16.321 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:19:16.321 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:19:16.321 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:19:16.321 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:16.321 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:19:16.321 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:19:16.556 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:19:17.144 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:07:32 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:19:17.303 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:19:17.304 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T16:19:17.304 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T16:19:17.304 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T16:19:17.304 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:19:17.304 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:19:17.305 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:19:17.305 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:19:17.305 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-25T16:19:17.305 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T16:19:17.305 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:19:17.305 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 1.4 M 2024-03-25T16:19:17.305 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 6.4 M 2024-03-25T16:19:17.305 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:19:19.371 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 707 kB/s | 1.4 MB 00:02 2024-03-25T16:19:19.372 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:19.372 INFO:teuthology.orchestra.run.smithi018.stdout:Total 706 kB/s | 1.4 MB 00:02 2024-03-25T16:19:19.372 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:19:19.377 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:19:19.377 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:19:19.423 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:19:19.424 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:19:19.690 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:19:19.797 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:19:20.220 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:19:20.248 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 627 kB/s | 3.1 MB 00:05 2024-03-25T16:19:20.249 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:20.249 INFO:teuthology.orchestra.run.smithi169.stdout:Total 626 kB/s | 3.1 MB 00:05 2024-03-25T16:19:20.249 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:19:20.261 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:19:20.261 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:19:20.338 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:19:20.339 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:19:20.512 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:19:20.512 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:19:20.512 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:19:20.512 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:20.512 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:19:20.512 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:19:20.705 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:19:20.768 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:19:20.815 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:20.839 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:20.840 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:19:20.840 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:19:20.840 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:19:20.840 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:19:20.840 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:21.280 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:07:36 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:19:21.435 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:19:21.436 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:19:21.436 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T16:19:21.436 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:19:21.436 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:19:21.436 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:19:21.436 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:19:21.436 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:19:21.437 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:19:21.437 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T16:19:21.437 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:19:21.437 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 176 M 2024-03-25T16:19:21.437 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 753 M 2024-03-25T16:19:21.437 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:19:22.026 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:22.026 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:22.026 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:19:22.026 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:22.027 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:22.027 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:19:22.298 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-nbd 2024-03-25T16:19:22.886 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:40 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:19:23.037 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:19:23.037 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:23.038 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-03-25T16:19:23.038 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:23.038 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:19:23.038 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:19:23.038 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:23.038 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:19:23.038 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:23.038 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T16:19:23.038 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:23.038 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 171 k 2024-03-25T16:19:23.039 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 494 k 2024-03-25T16:19:23.039 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:19:23.735 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 246 kB/s | 171 kB 00:00 2024-03-25T16:19:23.736 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:23.736 INFO:teuthology.orchestra.run.smithi169.stdout:Total 246 kB/s | 171 kB 00:00 2024-03-25T16:19:23.736 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:19:23.744 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:19:23.745 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:19:23.803 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:19:23.803 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:19:23.949 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:19:24.067 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:24.505 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:24.789 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:19:24.790 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:24.790 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:19:24.790 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:24.790 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:24.790 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:19:25.019 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:19:25.596 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:43 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:19:25.750 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:19:25.751 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:25.751 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T16:19:25.751 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:25.751 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:19:25.751 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:19:25.751 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-03-25T16:19:25.751 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:19:25.751 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:25.751 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:19:25.751 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:19:25.751 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2024-03-25T16:19:25.751 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:19:25.752 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 91 M 2024-03-25T16:19:25.752 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 498 M 2024-03-25T16:19:25.752 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:20:07.584 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 549 kB/s | 22 MB 00:41 2024-03-25T16:21:11.492 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 660 kB/s | 68 MB 01:45 2024-03-25T16:21:11.493 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:11.493 INFO:teuthology.orchestra.run.smithi169.stdout:Total 877 kB/s | 91 MB 01:45 2024-03-25T16:21:11.494 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:21:11.497 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:21:11.497 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:21:12.087 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:21:12.087 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:21:15.199 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:21:22.392 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:21:22.533 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:21:25.886 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:21:25.886 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:21:26.145 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:21:26.145 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:21:26.145 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:21:26.145 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:26.145 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:26.145 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:21:26.146 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:21:26.358 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:21:26.927 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:04:44 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:21:27.077 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:21:27.078 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:21:27.078 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T16:21:27.078 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:21:27.078 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:21:27.078 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:21:27.078 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:21:27.078 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:21:27.078 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:21:27.078 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T16:21:27.079 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:21:27.079 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 416 M 2024-03-25T16:21:27.079 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 1.7 G 2024-03-25T16:21:27.079 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:23:29.215 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 726 kB/s | 176 MB 04:07 2024-03-25T16:23:29.215 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:29.215 INFO:teuthology.orchestra.run.smithi018.stdout:Total 726 kB/s | 176 MB 04:07 2024-03-25T16:23:29.216 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:23:29.218 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:23:29.219 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:23:30.327 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:23:30.327 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:23:49.402 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:23:49.504 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:23:53.954 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:23:54.217 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:23:54.217 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:23:54.217 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:23:54.217 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:54.217 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:23:54.217 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:23:54.442 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:23:55.021 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:12:10 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:23:55.175 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:23:55.176 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:23:55.176 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T16:23:55.176 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:23:55.176 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:23:55.176 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:23:55.176 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:23:55.176 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:23:55.176 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:23:55.177 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T16:23:55.177 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:23:55.177 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 672 M 2024-03-25T16:23:55.177 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 3.1 G 2024-03-25T16:23:55.177 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:44:29.633 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 308 kB/s | 416 MB 23:02 2024-03-25T16:44:29.657 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:44:29.657 INFO:teuthology.orchestra.run.smithi169.stdout:Total 308 kB/s | 416 MB 23:02 2024-03-25T16:44:29.657 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:44:29.658 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:44:29.658 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:44:32.117 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:44:32.118 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:45:14.340 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:45:14.436 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:45:21.499 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:45:21.759 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:45:21.759 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:45:21.759 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:45:21.759 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:45:21.760 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:45:21.760 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:45:22.017 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:45:22.611 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:28:40 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:45:22.773 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:45:22.773 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T16:45:22.774 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T16:45:22.774 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T16:45:22.774 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:45:22.774 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:45:22.774 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:45:22.774 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:45:22.774 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-03-25T16:45:22.774 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T16:45:22.774 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:45:22.774 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 1.4 M 2024-03-25T16:45:22.775 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 6.4 M 2024-03-25T16:45:22.775 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:45:29.754 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 209 kB/s | 1.4 MB 00:06 2024-03-25T16:45:29.755 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:45:29.755 INFO:teuthology.orchestra.run.smithi169.stdout:Total 209 kB/s | 1.4 MB 00:06 2024-03-25T16:45:29.756 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:45:29.760 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:45:29.760 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:45:29.805 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:45:29.806 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:45:30.065 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:45:30.170 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:45:30.587 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:45:30.857 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:45:30.857 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:45:30.857 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:45:30.857 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:45:30.857 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:45:30.857 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:45:31.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:45:31.613 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:28:49 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:45:31.764 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:45:31.765 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:45:31.765 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T16:45:31.765 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:45:31.765 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:45:31.765 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:45:31.765 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:45:31.765 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:45:31.765 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:45:31.765 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T16:45:31.766 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:45:31.766 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 176 M 2024-03-25T16:45:31.766 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 753 M 2024-03-25T16:45:31.766 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:55:19.406 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 365 kB/s | 672 MB 31:24 2024-03-25T16:55:19.406 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:55:19.406 INFO:teuthology.orchestra.run.smithi018.stdout:Total 365 kB/s | 672 MB 31:24 2024-03-25T16:55:19.406 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:55:19.411 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:55:19.411 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:55:23.563 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:55:23.563 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:56:23.140 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 276 kB/s | 176 MB 10:51 2024-03-25T16:56:23.141 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:56:23.141 INFO:teuthology.orchestra.run.smithi169.stdout:Total 276 kB/s | 176 MB 10:51 2024-03-25T16:56:23.141 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T16:56:23.145 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T16:56:23.145 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T16:56:24.208 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T16:56:24.208 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T16:56:38.528 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:56:39.126 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:56:42.490 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T16:56:42.677 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:56:47.199 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:56:47.461 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:56:47.461 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:56:47.461 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:56:47.461 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:56:47.461 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:56:47.461 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:56:47.487 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:56:47.747 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:56:47.747 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:56:47.747 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T16:56:47.747 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:56:47.748 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:56:47.748 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T16:56:47.785 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:56:47.969 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:56:48.365 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:45:03 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:56:48.464 INFO:teuthology.orchestra.run.smithi018.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:56:48.519 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:56:48.520 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-25T16:56:48.520 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:56:48.532 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:40:06 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T16:56:48.591 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T16:56:48.680 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T16:56:48.681 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:56:48.681 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T16:56:48.681 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:56:48.681 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T16:56:48.681 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:56:48.681 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:56:48.681 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T16:56:48.681 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T16:56:48.681 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T16:56:48.681 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T16:56:48.681 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 672 M 2024-03-25T16:56:48.682 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 3.1 G 2024-03-25T16:56:48.682 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T16:56:49.167 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:45:04 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:56:49.320 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:56:49.321 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:56:49.321 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T16:56:49.321 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:56:49.321 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:56:49.321 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:56:49.321 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:56:49.321 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:56:49.321 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:56:49.321 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T16:56:49.321 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:56:49.322 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 20 M 2024-03-25T16:56:49.322 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 103 M 2024-03-25T16:56:49.322 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:57:17.053 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 727 kB/s | 20 MB 00:27 2024-03-25T16:57:17.054 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:57:17.054 INFO:teuthology.orchestra.run.smithi018.stdout:Total 727 kB/s | 20 MB 00:27 2024-03-25T16:57:17.055 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:57:17.059 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:57:17.059 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:57:17.215 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:57:17.216 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:57:19.579 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:57:19.688 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:57:20.617 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:57:20.890 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:57:20.890 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:57:20.890 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:57:20.890 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:57:20.891 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:57:20.891 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:57:21.075 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T16:57:21.652 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:45:36 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:57:21.807 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:57:21.807 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:57:21.807 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T16:57:21.808 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:57:21.808 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:57:21.808 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T16:57:21.808 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:57:21.808 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:57:21.808 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:57:21.808 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T16:57:21.808 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:57:21.809 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 48 M 2024-03-25T16:57:21.809 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 173 M 2024-03-25T16:57:21.809 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T16:59:09.575 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 453 kB/s | 48 MB 01:47 2024-03-25T16:59:09.575 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:59:09.575 INFO:teuthology.orchestra.run.smithi018.stdout:Total 453 kB/s | 48 MB 01:47 2024-03-25T16:59:09.575 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T16:59:09.578 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T16:59:09.578 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T16:59:09.904 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T16:59:09.905 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T16:59:14.934 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T16:59:15.032 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:59:16.389 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:59:16.654 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:59:16.654 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:59:16.654 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T16:59:16.654 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:59:16.654 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:59:16.655 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T16:59:16.830 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T16:59:17.406 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:47:32 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T16:59:17.561 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T16:59:17.562 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:59:17.562 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T16:59:17.562 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:59:17.562 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T16:59:17.563 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T16:59:17.563 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:59:17.563 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T16:59:17.563 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T16:59:17.563 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T16:59:17.563 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T16:59:17.563 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 99 M 2024-03-25T16:59:17.563 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 449 M 2024-03-25T16:59:17.563 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T17:05:13.357 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 284 kB/s | 99 MB 05:55 2024-03-25T17:05:13.357 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:05:13.357 INFO:teuthology.orchestra.run.smithi018.stdout:Total 284 kB/s | 99 MB 05:55 2024-03-25T17:05:13.357 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T17:05:13.361 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T17:05:13.361 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T17:05:13.999 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T17:05:13.999 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T17:05:24.870 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T17:05:24.973 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:05:27.934 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:05:28.198 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:05:28.199 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:05:28.199 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T17:05:28.199 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:05:28.199 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:05:28.199 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T17:05:28.503 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:05:29.082 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:53:44 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T17:05:29.235 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T17:05:29.236 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:05:29.236 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T17:05:29.236 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:05:29.236 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T17:05:29.236 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:05:29.237 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:05:29.237 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T17:05:29.237 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:05:29.237 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T17:05:29.237 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:05:29.237 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 287 M 2024-03-25T17:05:29.237 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 1.2 G 2024-03-25T17:05:29.237 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T17:21:03.245 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 473 kB/s | 672 MB 24:14 2024-03-25T17:21:03.265 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:21:03.265 INFO:teuthology.orchestra.run.smithi169.stdout:Total 473 kB/s | 672 MB 24:14 2024-03-25T17:21:03.265 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T17:21:03.266 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T17:21:03.266 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T17:21:07.232 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T17:21:07.233 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T17:22:18.154 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T17:22:18.755 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:22:27.935 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:22:28.196 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:22:28.196 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:22:28.196 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T17:22:28.196 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:22:28.196 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:22:28.196 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:22:28.490 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:22:29.057 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:05:47 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T17:22:29.153 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:22:29.205 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T17:22:29.206 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-03-25T17:22:29.206 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:22:29.280 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:22:29.885 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:05:47 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T17:22:30.042 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T17:22:30.043 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:22:30.043 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T17:22:30.043 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:22:30.043 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T17:22:30.043 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:22:30.043 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:22:30.043 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T17:22:30.043 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:22:30.043 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T17:22:30.043 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:22:30.044 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 20 M 2024-03-25T17:22:30.044 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 103 M 2024-03-25T17:22:30.044 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T17:23:34.348 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 314 kB/s | 20 MB 01:04 2024-03-25T17:23:34.348 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:23:34.348 INFO:teuthology.orchestra.run.smithi169.stdout:Total 314 kB/s | 20 MB 01:04 2024-03-25T17:23:34.349 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T17:23:34.352 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T17:23:34.352 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T17:23:34.502 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T17:23:34.502 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T17:23:36.737 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T17:23:36.911 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:23:37.914 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:23:38.168 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:23:38.168 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:23:38.168 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T17:23:38.168 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:23:38.168 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:23:38.168 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:23:38.368 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:23:38.939 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:06:56 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T17:23:39.088 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T17:23:39.088 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:23:39.088 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T17:23:39.088 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:23:39.088 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T17:23:39.088 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:23:39.088 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:23:39.088 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T17:23:39.089 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:23:39.089 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T17:23:39.089 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:23:39.089 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 48 M 2024-03-25T17:23:39.089 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 173 M 2024-03-25T17:23:39.089 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T17:24:11.322 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 262 kB/s | 287 MB 18:42 2024-03-25T17:24:11.323 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:24:11.323 INFO:teuthology.orchestra.run.smithi018.stdout:Total 262 kB/s | 287 MB 18:42 2024-03-25T17:24:11.323 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T17:24:11.326 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T17:24:11.327 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T17:24:13.118 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T17:24:13.118 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T17:24:44.218 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T17:24:44.320 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:24:51.695 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:24:52.023 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:24:52.023 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:24:52.023 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T17:24:52.023 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:24:52.023 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:24:52.023 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T17:24:52.257 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:24:52.836 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:13:07 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T17:24:52.992 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T17:24:52.993 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:24:52.993 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T17:24:52.993 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:24:52.993 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T17:24:52.993 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:24:52.993 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:24:52.993 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T17:24:52.993 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:24:52.993 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T17:24:52.993 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:24:52.993 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 5.9 M 2024-03-25T17:24:52.993 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 28 M 2024-03-25T17:24:52.994 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T17:25:08.135 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 402 kB/s | 5.9 MB 00:15 2024-03-25T17:25:08.135 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:25:08.136 INFO:teuthology.orchestra.run.smithi018.stdout:Total 402 kB/s | 5.9 MB 00:15 2024-03-25T17:25:08.136 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T17:25:08.140 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T17:25:08.140 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T17:25:08.215 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T17:25:08.215 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T17:25:08.996 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T17:25:09.094 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:25:09.645 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:25:09.912 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:25:09.912 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:25:09.912 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T17:25:09.912 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:25:09.912 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:25:09.912 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T17:25:10.103 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:25:10.677 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:13:25 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T17:25:10.835 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T17:25:10.836 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:25:10.836 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T17:25:10.836 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:25:10.837 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T17:25:10.837 INFO:teuthology.orchestra.run.smithi018.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:25:10.837 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:25:10.837 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T17:25:10.837 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:25:10.837 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T17:25:10.837 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:25:10.837 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 109 k 2024-03-25T17:25:10.837 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 440 k 2024-03-25T17:25:10.837 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T17:25:11.538 INFO:teuthology.orchestra.run.smithi018.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 155 kB/s | 109 kB 00:00 2024-03-25T17:25:11.538 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:25:11.538 INFO:teuthology.orchestra.run.smithi018.stdout:Total 155 kB/s | 109 kB 00:00 2024-03-25T17:25:11.539 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T17:25:11.542 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T17:25:11.542 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T17:25:11.578 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T17:25:11.579 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T17:25:11.695 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T17:25:11.814 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:25:12.196 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:25:12.468 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:25:12.469 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:25:12.469 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T17:25:12.469 INFO:teuthology.orchestra.run.smithi018.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:25:12.469 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:25:12.469 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T17:25:12.675 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:25:13.250 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:13:28 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T17:25:13.405 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T17:25:13.406 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:25:13.407 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T17:25:13.407 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:25:13.407 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T17:25:13.407 INFO:teuthology.orchestra.run.smithi018.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:25:13.407 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:25:13.407 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T17:25:13.407 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:25:13.407 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T17:25:13.407 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:25:13.407 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 5.4 M 2024-03-25T17:25:13.408 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 25 M 2024-03-25T17:25:13.408 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T17:25:35.121 INFO:teuthology.orchestra.run.smithi018.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 5.4 MB 00:21 2024-03-25T17:25:35.122 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:25:35.122 INFO:teuthology.orchestra.run.smithi018.stdout:Total 252 kB/s | 5.4 MB 00:21 2024-03-25T17:25:35.123 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T17:25:35.127 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T17:25:35.127 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T17:25:35.198 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T17:25:35.198 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T17:25:35.903 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T17:25:36.023 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:25:36.531 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:25:36.798 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:25:36.799 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:25:36.799 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T17:25:36.799 INFO:teuthology.orchestra.run.smithi018.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:25:36.799 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:25:36.799 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T17:25:36.996 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install librados2-debuginfo 2024-03-25T17:25:37.572 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:13:52 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T17:25:37.728 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T17:25:37.729 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:25:37.729 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T17:25:37.730 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:25:37.730 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T17:25:37.730 INFO:teuthology.orchestra.run.smithi018.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:25:37.730 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:25:37.730 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T17:25:37.730 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:25:37.730 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T17:25:37.730 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:25:37.730 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 44 M 2024-03-25T17:25:37.730 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 253 M 2024-03-25T17:25:37.731 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T17:26:04.096 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 337 kB/s | 48 MB 02:25 2024-03-25T17:26:04.097 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:26:04.097 INFO:teuthology.orchestra.run.smithi169.stdout:Total 337 kB/s | 48 MB 02:25 2024-03-25T17:26:04.097 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T17:26:04.100 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T17:26:04.101 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T17:26:04.415 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T17:26:04.416 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T17:26:09.170 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T17:26:09.260 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:26:10.629 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:26:10.878 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:26:10.878 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:26:10.878 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T17:26:10.878 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:26:10.878 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:26:10.878 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:26:11.083 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:26:11.650 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:09:29 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T17:26:11.799 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T17:26:11.799 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:26:11.800 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T17:26:11.800 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:26:11.800 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T17:26:11.800 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:26:11.800 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:26:11.800 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T17:26:11.800 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:26:11.800 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T17:26:11.800 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:26:11.800 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 99 M 2024-03-25T17:26:11.800 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 449 M 2024-03-25T17:26:11.800 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T17:27:50.480 INFO:teuthology.orchestra.run.smithi018.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 342 kB/s | 44 MB 02:12 2024-03-25T17:27:50.480 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:27:50.481 INFO:teuthology.orchestra.run.smithi018.stdout:Total 342 kB/s | 44 MB 02:12 2024-03-25T17:27:50.481 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T17:27:50.484 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T17:27:50.484 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T17:27:50.798 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T17:27:50.799 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T17:27:56.145 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T17:27:56.256 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:27:58.105 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:27:58.369 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:27:58.369 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:27:58.369 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T17:27:58.370 INFO:teuthology.orchestra.run.smithi018.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:27:58.370 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:27:58.370 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T17:27:58.569 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:27:59.147 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:16:14 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T17:27:59.303 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T17:27:59.303 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:27:59.303 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T17:27:59.304 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:27:59.304 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T17:27:59.304 INFO:teuthology.orchestra.run.smithi018.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:27:59.304 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:27:59.304 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T17:27:59.304 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:27:59.304 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T17:27:59.304 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:27:59.305 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 52 M 2024-03-25T17:27:59.305 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 308 M 2024-03-25T17:27:59.305 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T17:30:37.302 INFO:teuthology.orchestra.run.smithi018.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 336 kB/s | 52 MB 02:37 2024-03-25T17:30:37.302 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:30:37.302 INFO:teuthology.orchestra.run.smithi018.stdout:Total 336 kB/s | 52 MB 02:37 2024-03-25T17:30:37.303 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T17:30:37.306 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T17:30:37.306 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T17:30:37.664 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T17:30:37.665 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T17:30:43.999 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T17:30:44.110 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:30:46.293 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:30:46.560 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:30:46.561 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:30:46.561 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T17:30:46.561 INFO:teuthology.orchestra.run.smithi018.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:30:46.561 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:30:46.561 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T17:30:46.872 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T17:30:47.444 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:19:02 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T17:30:47.599 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T17:30:47.600 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:30:47.600 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T17:30:47.600 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:30:47.601 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T17:30:47.601 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T17:30:47.601 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:30:47.601 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T17:30:47.601 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:30:47.601 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T17:30:47.601 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:30:47.601 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 615 k 2024-03-25T17:30:47.601 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 2.1 M 2024-03-25T17:30:47.601 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T17:30:49.220 INFO:teuthology.orchestra.run.smithi018.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 380 kB/s | 615 kB 00:01 2024-03-25T17:30:49.220 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:30:49.221 INFO:teuthology.orchestra.run.smithi018.stdout:Total 380 kB/s | 615 kB 00:01 2024-03-25T17:30:49.221 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T17:30:49.225 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T17:30:49.225 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T17:30:49.269 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T17:30:49.269 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T17:30:49.468 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T17:30:49.564 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:30:50.069 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:30:50.344 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:30:50.344 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:30:50.344 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T17:30:50.344 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:30:50.344 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:30:50.344 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T17:30:50.669 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-rados-debuginfo 2024-03-25T17:30:51.245 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:19:06 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T17:30:51.401 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T17:30:51.402 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:30:51.402 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T17:30:51.402 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:30:51.402 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T17:30:51.402 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T17:30:51.402 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:30:51.402 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T17:30:51.403 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:30:51.403 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T17:30:51.403 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:30:51.403 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 771 k 2024-03-25T17:30:51.403 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 2.7 M 2024-03-25T17:30:51.403 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T17:30:54.556 INFO:teuthology.orchestra.run.smithi018.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 244 kB/s | 771 kB 00:03 2024-03-25T17:30:54.556 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:30:54.557 INFO:teuthology.orchestra.run.smithi018.stdout:Total 244 kB/s | 771 kB 00:03 2024-03-25T17:30:54.557 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T17:30:54.561 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T17:30:54.561 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T17:30:54.605 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T17:30:54.605 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T17:30:54.792 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T17:30:54.907 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:30:55.292 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:30:55.561 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:30:55.561 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:30:55.562 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T17:30:55.562 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:30:55.562 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:30:55.562 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T17:30:55.797 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T17:30:56.376 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:19:11 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T17:30:56.530 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T17:30:56.531 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:30:56.531 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T17:30:56.531 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:30:56.532 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T17:30:56.532 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T17:30:56.532 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:30:56.532 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T17:30:56.532 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:30:56.532 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T17:30:56.532 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:30:56.532 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 1.1 M 2024-03-25T17:30:56.532 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 4.3 M 2024-03-25T17:30:56.532 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T17:31:00.342 INFO:teuthology.orchestra.run.smithi018.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 302 kB/s | 1.1 MB 00:03 2024-03-25T17:31:00.343 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:31:00.343 INFO:teuthology.orchestra.run.smithi018.stdout:Total 302 kB/s | 1.1 MB 00:03 2024-03-25T17:31:00.343 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T17:31:00.347 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T17:31:00.347 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T17:31:00.393 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T17:31:00.393 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T17:31:00.624 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T17:31:00.736 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:31:01.171 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:31:01.444 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:31:01.444 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:31:01.444 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T17:31:01.444 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:31:01.444 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:31:01.445 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T17:31:01.652 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T17:31:02.230 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:19:17 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T17:31:02.390 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T17:31:02.390 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:31:02.390 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T17:31:02.390 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:31:02.391 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T17:31:02.391 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T17:31:02.391 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:31:02.391 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T17:31:02.391 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:31:02.391 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T17:31:02.391 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:31:02.391 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 337 k 2024-03-25T17:31:02.391 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 1.0 M 2024-03-25T17:31:02.391 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T17:31:04.020 INFO:teuthology.orchestra.run.smithi018.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 207 kB/s | 337 kB 00:01 2024-03-25T17:31:04.020 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:31:04.021 INFO:teuthology.orchestra.run.smithi018.stdout:Total 207 kB/s | 337 kB 00:01 2024-03-25T17:31:04.021 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T17:31:04.025 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T17:31:04.025 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T17:31:04.066 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T17:31:04.067 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T17:31:04.212 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T17:31:04.314 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:31:04.679 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:31:04.953 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:31:04.953 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:31:04.953 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T17:31:04.953 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:31:04.953 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:31:04.953 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T17:31:05.155 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T17:31:05.729 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:19:20 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T17:31:05.884 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T17:31:05.885 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:31:05.885 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T17:31:05.886 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:31:05.886 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T17:31:05.886 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T17:31:05.886 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:31:05.886 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T17:31:05.886 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:31:05.886 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T17:31:05.886 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:31:05.886 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 509 k 2024-03-25T17:31:05.886 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 2.5 M 2024-03-25T17:31:05.886 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T17:31:08.748 INFO:teuthology.orchestra.run.smithi018.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 178 kB/s | 509 kB 00:02 2024-03-25T17:31:08.748 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:31:08.749 INFO:teuthology.orchestra.run.smithi018.stdout:Total 178 kB/s | 509 kB 00:02 2024-03-25T17:31:08.749 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T17:31:08.753 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T17:31:08.753 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T17:31:08.794 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T17:31:08.795 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T17:31:08.959 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T17:31:09.057 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:31:09.430 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:31:09.699 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:31:09.699 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:31:09.700 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T17:31:09.700 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:31:09.700 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:31:09.700 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T17:31:09.887 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T17:31:10.464 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:19:25 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T17:31:10.621 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T17:31:10.621 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:31:10.621 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T17:31:10.622 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:31:10.622 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T17:31:10.622 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:31:10.622 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:31:10.622 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T17:31:10.622 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:31:10.622 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T17:31:10.622 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:31:10.622 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 52 M 2024-03-25T17:31:10.622 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 298 M 2024-03-25T17:31:10.622 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T17:32:01.349 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 289 kB/s | 99 MB 05:49 2024-03-25T17:32:01.350 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:32:01.350 INFO:teuthology.orchestra.run.smithi169.stdout:Total 289 kB/s | 99 MB 05:49 2024-03-25T17:32:01.350 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T17:32:01.354 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T17:32:01.354 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T17:32:01.970 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T17:32:01.970 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T17:32:12.293 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T17:32:12.397 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:32:15.306 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:32:15.577 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:32:15.578 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:32:15.578 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T17:32:15.578 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:32:15.578 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:32:15.578 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:32:15.797 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:32:16.369 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:15:34 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T17:32:16.518 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T17:32:16.519 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:32:16.519 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T17:32:16.519 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:32:16.519 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T17:32:16.519 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:32:16.519 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:32:16.519 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T17:32:16.519 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:32:16.519 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T17:32:16.519 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:32:16.520 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 287 M 2024-03-25T17:32:16.520 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 1.2 G 2024-03-25T17:32:16.520 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T17:33:04.509 INFO:teuthology.orchestra.run.smithi018.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 463 kB/s | 52 MB 01:53 2024-03-25T17:33:04.510 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:33:04.510 INFO:teuthology.orchestra.run.smithi018.stdout:Total 463 kB/s | 52 MB 01:53 2024-03-25T17:33:04.510 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T17:33:04.514 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T17:33:04.514 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T17:33:04.865 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T17:33:04.865 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T17:33:11.120 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T17:33:11.226 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:33:13.251 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:33:13.514 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:33:13.514 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:33:13.514 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T17:33:13.514 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:33:13.514 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:33:13.514 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T17:33:13.694 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T17:33:14.274 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 1:21:29 ago on Mon 25 Mar 2024 04:11:45 PM UTC. 2024-03-25T17:33:14.429 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-25T17:33:14.430 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:33:14.430 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-25T17:33:14.430 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:33:14.430 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-25T17:33:14.430 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T17:33:14.430 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:33:14.430 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-25T17:33:14.430 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-25T17:33:14.430 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-25T17:33:14.430 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:33:14.430 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 1.3 M 2024-03-25T17:33:14.430 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 5.5 M 2024-03-25T17:33:14.431 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-25T17:33:20.010 INFO:teuthology.orchestra.run.smithi018.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 230 kB/s | 1.3 MB 00:05 2024-03-25T17:33:20.011 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:33:20.011 INFO:teuthology.orchestra.run.smithi018.stdout:Total 230 kB/s | 1.3 MB 00:05 2024-03-25T17:33:20.012 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-25T17:33:20.016 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-25T17:33:20.016 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-25T17:33:20.062 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-25T17:33:20.063 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-25T17:33:20.312 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-25T17:33:20.404 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:33:20.794 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:33:21.070 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:33:21.070 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:33:21.070 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-25T17:33:21.070 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:33:21.070 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:33:21.070 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-25T17:33:21.243 DEBUG:teuthology.parallel:result is None 2024-03-25T17:47:58.733 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 312 kB/s | 287 MB 15:42 2024-03-25T17:47:58.733 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:47:58.733 INFO:teuthology.orchestra.run.smithi169.stdout:Total 312 kB/s | 287 MB 15:42 2024-03-25T17:47:58.734 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T17:47:58.736 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T17:47:58.736 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T17:48:00.445 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T17:48:00.445 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T17:48:29.993 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T17:48:30.286 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:37.978 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:38.240 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:38.240 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:48:38.240 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T17:48:38.240 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:38.240 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:48:38.240 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:48:38.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:48:39.043 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:31:57 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T17:48:39.194 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T17:48:39.195 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:48:39.195 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T17:48:39.195 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:48:39.195 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T17:48:39.195 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:48:39.195 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:48:39.195 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T17:48:39.195 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:48:39.195 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T17:48:39.195 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:48:39.196 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 5.9 M 2024-03-25T17:48:39.196 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 28 M 2024-03-25T17:48:39.196 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T17:48:52.404 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 461 kB/s | 5.9 MB 00:13 2024-03-25T17:48:52.405 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:48:52.405 INFO:teuthology.orchestra.run.smithi169.stdout:Total 461 kB/s | 5.9 MB 00:13 2024-03-25T17:48:52.405 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T17:48:52.409 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T17:48:52.410 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T17:48:52.482 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T17:48:52.482 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T17:48:53.225 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T17:48:53.327 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:48:53.950 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:48:54.206 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:48:54.206 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:48:54.206 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T17:48:54.206 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:54.206 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:48:54.206 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:48:54.415 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:48:54.984 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:32:12 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T17:48:55.134 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T17:48:55.135 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:48:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T17:48:55.135 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:48:55.135 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T17:48:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:48:55.135 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:48:55.135 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T17:48:55.136 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:48:55.136 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T17:48:55.136 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:48:55.136 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 109 k 2024-03-25T17:48:55.136 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 440 k 2024-03-25T17:48:55.136 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T17:48:55.824 INFO:teuthology.orchestra.run.smithi169.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 158 kB/s | 109 kB 00:00 2024-03-25T17:48:55.824 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:48:55.825 INFO:teuthology.orchestra.run.smithi169.stdout:Total 158 kB/s | 109 kB 00:00 2024-03-25T17:48:55.825 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T17:48:55.828 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T17:48:55.828 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T17:48:55.863 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T17:48:55.863 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T17:48:55.983 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T17:48:56.081 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:48:56.484 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:48:56.738 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:48:56.739 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:48:56.739 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T17:48:56.739 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:56.739 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:48:56.739 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:48:56.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:48:57.509 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:32:15 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T17:48:57.657 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T17:48:57.657 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:48:57.657 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T17:48:57.657 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:48:57.658 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T17:48:57.658 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:48:57.658 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:48:57.658 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T17:48:57.658 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:48:57.658 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T17:48:57.658 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:48:57.658 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 5.4 M 2024-03-25T17:48:57.659 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 25 M 2024-03-25T17:48:57.659 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T17:49:15.441 INFO:teuthology.orchestra.run.smithi169.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 308 kB/s | 5.4 MB 00:17 2024-03-25T17:49:15.442 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:15.442 INFO:teuthology.orchestra.run.smithi169.stdout:Total 308 kB/s | 5.4 MB 00:17 2024-03-25T17:49:15.442 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T17:49:15.446 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T17:49:15.446 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T17:49:15.516 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T17:49:15.516 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T17:49:16.186 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T17:49:16.289 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:49:16.869 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:49:17.126 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:49:17.126 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:49:17.126 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T17:49:17.126 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:17.126 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:49:17.126 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:49:17.350 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados2-debuginfo 2024-03-25T17:49:17.921 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:32:35 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T17:49:18.072 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T17:49:18.073 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:49:18.073 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T17:49:18.073 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:49:18.073 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T17:49:18.073 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:49:18.073 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:49:18.073 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T17:49:18.073 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:49:18.073 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T17:49:18.073 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:49:18.074 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 44 M 2024-03-25T17:49:18.074 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 253 M 2024-03-25T17:49:18.074 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T17:51:01.500 INFO:teuthology.orchestra.run.smithi169.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 438 kB/s | 44 MB 01:43 2024-03-25T17:51:01.500 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:01.500 INFO:teuthology.orchestra.run.smithi169.stdout:Total 438 kB/s | 44 MB 01:43 2024-03-25T17:51:01.500 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T17:51:01.503 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T17:51:01.503 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T17:51:01.803 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T17:51:01.803 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T17:51:06.873 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T17:51:06.980 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:51:08.899 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:51:09.157 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:51:09.157 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:51:09.158 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T17:51:09.158 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:09.158 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:51:09.158 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:51:09.367 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:51:09.935 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:34:27 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T17:51:10.087 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T17:51:10.088 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:51:10.088 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T17:51:10.088 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:51:10.089 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T17:51:10.089 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:51:10.089 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:51:10.089 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T17:51:10.089 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:51:10.089 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T17:51:10.089 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:51:10.089 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 52 M 2024-03-25T17:51:10.089 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 308 M 2024-03-25T17:51:10.089 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T17:52:32.425 INFO:teuthology.orchestra.run.smithi169.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 645 kB/s | 52 MB 01:22 2024-03-25T17:52:32.426 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:32.426 INFO:teuthology.orchestra.run.smithi169.stdout:Total 645 kB/s | 52 MB 01:22 2024-03-25T17:52:32.426 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T17:52:32.429 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T17:52:32.429 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T17:52:32.772 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T17:52:32.773 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T17:52:38.808 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T17:52:38.980 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:41.188 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:41.444 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:41.444 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:52:41.444 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T17:52:41.444 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:41.444 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:52:41.445 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:52:41.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T17:52:42.224 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:36:00 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T17:52:42.373 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T17:52:42.373 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:52:42.374 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T17:52:42.374 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:52:42.374 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T17:52:42.374 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T17:52:42.374 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:52:42.374 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T17:52:42.374 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:52:42.374 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T17:52:42.374 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:52:42.374 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 615 k 2024-03-25T17:52:42.374 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 2.1 M 2024-03-25T17:52:42.374 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T17:52:44.318 INFO:teuthology.orchestra.run.smithi169.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 316 kB/s | 615 kB 00:01 2024-03-25T17:52:44.319 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:44.320 INFO:teuthology.orchestra.run.smithi169.stdout:Total 316 kB/s | 615 kB 00:01 2024-03-25T17:52:44.320 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T17:52:44.324 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T17:52:44.324 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T17:52:44.365 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T17:52:44.365 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T17:52:44.531 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T17:52:44.634 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:52:45.031 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:52:45.284 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:52:45.284 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:52:45.284 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T17:52:45.284 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:45.284 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:52:45.284 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:52:45.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rados-debuginfo 2024-03-25T17:52:46.064 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:36:04 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T17:52:46.215 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T17:52:46.216 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:52:46.216 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T17:52:46.216 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:52:46.216 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T17:52:46.216 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T17:52:46.216 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:52:46.216 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T17:52:46.217 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:52:46.217 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T17:52:46.217 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:52:46.217 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 771 k 2024-03-25T17:52:46.217 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 2.7 M 2024-03-25T17:52:46.217 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T17:52:51.279 INFO:teuthology.orchestra.run.smithi169.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 152 kB/s | 771 kB 00:05 2024-03-25T17:52:51.280 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:51.280 INFO:teuthology.orchestra.run.smithi169.stdout:Total 152 kB/s | 771 kB 00:05 2024-03-25T17:52:51.281 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T17:52:51.284 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T17:52:51.285 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T17:52:51.326 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T17:52:51.327 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T17:52:51.511 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T17:52:51.612 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:52:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:52:52.241 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:52:52.241 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:52:52.242 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T17:52:52.242 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:52.242 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:52:52.242 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:52:52.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T17:52:53.041 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:36:11 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T17:52:53.195 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T17:52:53.196 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:52:53.196 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T17:52:53.196 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:52:53.196 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T17:52:53.196 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T17:52:53.196 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:52:53.196 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T17:52:53.196 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:52:53.196 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T17:52:53.197 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:52:53.197 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 1.1 M 2024-03-25T17:52:53.197 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 4.3 M 2024-03-25T17:52:53.197 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T17:52:57.758 INFO:teuthology.orchestra.run.smithi169.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 252 kB/s | 1.1 MB 00:04 2024-03-25T17:52:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:Total 252 kB/s | 1.1 MB 00:04 2024-03-25T17:52:57.759 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T17:52:57.763 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T17:52:57.763 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T17:52:57.806 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T17:52:57.806 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T17:52:58.024 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T17:52:58.131 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:52:58.551 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:52:58.809 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:52:58.809 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:52:58.809 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T17:52:58.809 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:58.809 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:52:58.809 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:52:59.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T17:52:59.604 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:36:17 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T17:52:59.755 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T17:52:59.755 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:52:59.755 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T17:52:59.756 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:52:59.756 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T17:52:59.756 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T17:52:59.756 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:52:59.756 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T17:52:59.756 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:52:59.756 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T17:52:59.756 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:52:59.756 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 337 k 2024-03-25T17:52:59.756 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 1.0 M 2024-03-25T17:52:59.756 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T17:53:01.135 INFO:teuthology.orchestra.run.smithi169.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 245 kB/s | 337 kB 00:01 2024-03-25T17:53:01.136 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:01.136 INFO:teuthology.orchestra.run.smithi169.stdout:Total 244 kB/s | 337 kB 00:01 2024-03-25T17:53:01.136 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T17:53:01.140 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T17:53:01.140 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T17:53:01.181 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T17:53:01.181 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T17:53:01.326 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T17:53:01.442 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:53:01.833 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:53:02.088 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:53:02.089 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:53:02.089 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T17:53:02.089 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:02.089 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:53:02.089 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:53:02.355 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T17:53:02.918 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:36:20 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T17:53:03.067 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T17:53:03.068 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:53:03.068 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T17:53:03.068 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:53:03.068 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T17:53:03.068 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T17:53:03.068 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:53:03.068 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T17:53:03.068 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:53:03.068 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T17:53:03.068 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:53:03.069 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 509 k 2024-03-25T17:53:03.069 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 2.5 M 2024-03-25T17:53:03.069 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T17:53:06.905 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 133 kB/s | 509 kB 00:03 2024-03-25T17:53:06.905 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:06.906 INFO:teuthology.orchestra.run.smithi169.stdout:Total 133 kB/s | 509 kB 00:03 2024-03-25T17:53:06.906 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T17:53:06.910 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T17:53:06.910 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T17:53:06.951 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T17:53:06.951 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T17:53:07.113 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T17:53:07.212 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:07.596 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:07.856 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:07.857 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:53:07.857 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T17:53:07.857 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:07.857 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:53:07.857 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:53:08.097 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T17:53:08.664 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:36:26 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T17:53:08.813 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T17:53:08.814 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:53:08.814 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T17:53:08.814 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:53:08.814 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T17:53:08.815 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:53:08.815 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:53:08.815 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T17:53:08.815 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:53:08.815 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T17:53:08.815 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:53:08.815 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 52 M 2024-03-25T17:53:08.815 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 298 M 2024-03-25T17:53:08.815 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T17:55:12.234 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 428 kB/s | 52 MB 02:03 2024-03-25T17:55:12.235 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:12.235 INFO:teuthology.orchestra.run.smithi169.stdout:Total 428 kB/s | 52 MB 02:03 2024-03-25T17:55:12.235 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T17:55:12.238 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T17:55:12.238 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T17:55:12.574 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T17:55:12.574 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T17:55:18.475 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T17:55:18.765 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:55:20.858 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:55:21.111 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:55:21.111 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:21.111 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T17:55:21.111 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:21.111 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:21.111 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:55:21.353 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T17:55:21.928 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 1:38:39 ago on Mon 25 Mar 2024 04:16:42 PM UTC. 2024-03-25T17:55:22.077 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-03-25T17:55:22.078 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:55:22.078 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-03-25T17:55:22.078 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:55:22.078 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-03-25T17:55:22.078 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T17:55:22.078 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:22.078 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-03-25T17:55:22.078 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-03-25T17:55:22.079 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-03-25T17:55:22.079 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:22.079 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 1.3 M 2024-03-25T17:55:22.079 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 5.5 M 2024-03-25T17:55:22.079 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-03-25T17:55:26.726 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 276 kB/s | 1.3 MB 00:04 2024-03-25T17:55:26.727 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:26.727 INFO:teuthology.orchestra.run.smithi169.stdout:Total 276 kB/s | 1.3 MB 00:04 2024-03-25T17:55:26.727 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-03-25T17:55:26.731 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-03-25T17:55:26.732 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-03-25T17:55:26.777 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-03-25T17:55:26.778 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-03-25T17:55:27.017 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-03-25T17:55:27.117 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:55:27.545 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:55:27.809 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:55:27.809 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:27.809 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-03-25T17:55:27.810 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:27.810 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:27.810 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-03-25T17:55:28.027 DEBUG:teuthology.parallel:result is None 2024-03-25T17:55:28.028 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:55:28.176 DEBUG:teuthology.orchestra.run.smithi018:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T17:55:28.202 INFO:teuthology.orchestra.run.smithi018.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T17:55:28.202 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T17:55:28.203 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T17:55:28.204 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:55:28.351 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T17:55:28.372 INFO:teuthology.orchestra.run.smithi169.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T17:55:28.373 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T17:55:28.373 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T17:55:28.375 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T17:55:28.375 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:55:28.375 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T17:55:28.407 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T17:55:28.407 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T17:55:28.440 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T17:55:28.440 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:55:28.440 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T17:55:28.478 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T17:55:28.548 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T17:55:28.548 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T17:55:28.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T17:55:28.642 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T17:55:28.642 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:55:28.642 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T17:55:28.672 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T17:55:28.742 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T17:55:28.742 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T17:55:28.769 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T17:55:28.836 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T17:55:28.837 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:55:28.837 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T17:55:28.866 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T17:55:28.938 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T17:55:28.938 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T17:55:28.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T17:55:29.033 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T17:55:29.126 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T17:55:29.126 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 777 /var/log/ceph 2024-03-25T17:55:29.129 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 777 /var/log/ceph 2024-03-25T17:55:29.157 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T17:55:29.157 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T17:55:29.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T17:55:29.228 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T17:55:29.228 DEBUG:teuthology.orchestra.run.smithi018:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T17:55:29.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T17:55:29.327 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T17:55:29.328 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T17:55:29.328 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621241', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621241', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi018.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM5iihOi7mgKna/7JMuqrPgo58gKUIVlTPVjbWJCGi0Pig7Q+zNrm0Y3aQEAt+gWMd16I3LFzvsidKHn5+fpqlU=', 'smithi169.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGqazL4b6MWoRUTI63i1G8u0ySRHWPcHkfVSpRawEMRw0lsdpluWX2Sf71VruWn3bWzc89YLVmC5mUHvNm88rcQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T17:55:29.328 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T17:55:29.341 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T17:55:29.356 DEBUG:teuthology.orchestra.run.smithi018:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T17:55:29.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T17:55:29.424 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:55:29.425 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T17:55:29.481 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T17:55:29.481 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_1 2024-03-25T17:55:29.538 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T17:55:29.538 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:55:29.538 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T17:55:29.538 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:55:29.538 INFO:teuthology.orchestra.run.smithi018.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:55:29.538 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2024-03-25 16:13:51.242065989 +0000 2024-03-25T17:55:29.538 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2024-03-25 16:10:45.564937082 +0000 2024-03-25T17:55:29.538 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2024-03-25 16:10:45.564937082 +0000 2024-03-25T17:55:29.538 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: 2024-03-25 16:10:45.564937082 +0000 2024-03-25T17:55:29.539 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T17:55:29.606 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2024-03-25T17:55:29.606 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2024-03-25T17:55:29.606 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 9.9692e-05 s, 5.1 MB/s 2024-03-25T17:55:29.607 DEBUG:teuthology.orchestra.run.smithi018:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T17:55:29.665 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_2 2024-03-25T17:55:29.721 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T17:55:29.721 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:55:29.721 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T17:55:29.721 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:55:29.721 INFO:teuthology.orchestra.run.smithi018.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:55:29.721 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2024-03-25 16:13:51.242065989 +0000 2024-03-25T17:55:29.721 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2024-03-25 16:10:45.906933495 +0000 2024-03-25T17:55:29.721 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2024-03-25 16:10:45.906933495 +0000 2024-03-25T17:55:29.722 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: 2024-03-25 16:10:45.906933495 +0000 2024-03-25T17:55:29.722 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T17:55:29.789 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2024-03-25T17:55:29.789 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2024-03-25T17:55:29.789 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.000103504 s, 4.9 MB/s 2024-03-25T17:55:29.790 DEBUG:teuthology.orchestra.run.smithi018:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T17:55:29.847 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_3 2024-03-25T17:55:29.905 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T17:55:29.905 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:55:29.905 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T17:55:29.905 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:55:29.905 INFO:teuthology.orchestra.run.smithi018.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:55:29.905 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2024-03-25 16:13:51.243065979 +0000 2024-03-25T17:55:29.905 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2024-03-25 16:10:46.242929970 +0000 2024-03-25T17:55:29.905 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2024-03-25 16:10:46.242929970 +0000 2024-03-25T17:55:29.905 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: 2024-03-25 16:10:46.242929970 +0000 2024-03-25T17:55:29.906 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T17:55:29.972 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2024-03-25T17:55:29.972 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2024-03-25T17:55:29.972 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 9.9232e-05 s, 5.2 MB/s 2024-03-25T17:55:29.973 DEBUG:teuthology.orchestra.run.smithi018:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T17:55:30.029 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_4 2024-03-25T17:55:30.084 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T17:55:30.085 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:55:30.085 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T17:55:30.085 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:55:30.085 INFO:teuthology.orchestra.run.smithi018.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:55:30.085 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2024-03-25 16:13:51.243065979 +0000 2024-03-25T17:55:30.085 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2024-03-25 16:10:46.558926655 +0000 2024-03-25T17:55:30.085 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2024-03-25 16:10:46.558926655 +0000 2024-03-25T17:55:30.085 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: 2024-03-25 16:10:46.558926655 +0000 2024-03-25T17:55:30.086 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T17:55:30.150 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2024-03-25T17:55:30.150 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2024-03-25T17:55:30.151 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 9.7503e-05 s, 5.3 MB/s 2024-03-25T17:55:30.152 DEBUG:teuthology.orchestra.run.smithi018:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T17:55:30.208 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:55:30.208 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T17:55:30.208 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T17:55:30.226 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T17:55:30.226 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_1 2024-03-25T17:55:30.282 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T17:55:30.282 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:55:30.282 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T17:55:30.282 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:55:30.282 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:55:30.282 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-25 16:19:21.084986100 +0000 2024-03-25T17:55:30.282 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-25 16:10:49.215359966 +0000 2024-03-25T17:55:30.282 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-25 16:10:49.215359966 +0000 2024-03-25T17:55:30.282 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-25 16:10:49.215359966 +0000 2024-03-25T17:55:30.283 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T17:55:30.348 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-25T17:55:30.348 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-25T17:55:30.348 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000273823 s, 1.9 MB/s 2024-03-25T17:55:30.350 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T17:55:30.406 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_2 2024-03-25T17:55:30.462 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T17:55:30.463 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:55:30.463 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T17:55:30.463 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:55:30.463 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:55:30.463 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-25 16:19:21.085986079 +0000 2024-03-25T17:55:30.463 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-25 16:10:49.572352088 +0000 2024-03-25T17:55:30.463 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-25 16:10:49.572352088 +0000 2024-03-25T17:55:30.463 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-25 16:10:49.572352088 +0000 2024-03-25T17:55:30.463 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T17:55:30.530 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-25T17:55:30.530 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-25T17:55:30.530 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000177763 s, 2.9 MB/s 2024-03-25T17:55:30.531 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T17:55:30.588 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_3 2024-03-25T17:55:30.643 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T17:55:30.643 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:55:30.644 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T17:55:30.644 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:55:30.644 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:55:30.644 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-25 16:19:21.085986079 +0000 2024-03-25T17:55:30.644 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-25 16:10:49.915344519 +0000 2024-03-25T17:55:30.644 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-25 16:10:49.915344519 +0000 2024-03-25T17:55:30.644 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-25 16:10:49.915344519 +0000 2024-03-25T17:55:30.644 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T17:55:30.711 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-25T17:55:30.711 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-25T17:55:30.711 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000203458 s, 2.5 MB/s 2024-03-25T17:55:30.712 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T17:55:30.768 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_4 2024-03-25T17:55:30.823 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T17:55:30.823 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:55:30.823 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T17:55:30.823 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:55:30.823 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:55:30.823 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-03-25 16:19:21.085986079 +0000 2024-03-25T17:55:30.824 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-03-25 16:10:50.244337259 +0000 2024-03-25T17:55:30.824 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-03-25 16:10:50.244337259 +0000 2024-03-25T17:55:30.824 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-03-25 16:10:50.244337259 +0000 2024-03-25T17:55:30.824 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T17:55:30.888 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-03-25T17:55:30.889 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-03-25T17:55:30.889 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000193827 s, 2.6 MB/s 2024-03-25T17:55:30.890 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T17:55:30.945 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:55:30.946 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi018.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi169.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T17:55:30.946 INFO:tasks.ceph:Generating config... 2024-03-25T17:55:30.947 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T17:55:30.947 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T17:55:30.947 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T17:55:30.947 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T17:55:30.947 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T17:55:30.947 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T17:55:30.947 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T17:55:30.948 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T17:55:30.948 INFO:tasks.ceph:[global] ms type = async 2024-03-25T17:55:30.948 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T17:55:30.948 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T17:55:30.948 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T17:55:30.948 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T17:55:30.948 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T17:55:30.948 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T17:55:30.948 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T17:55:30.948 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T17:55:30.948 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T17:55:30.948 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T17:55:30.948 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T17:55:30.948 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T17:55:30.949 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T17:55:30.949 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T17:55:30.949 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T17:55:30.949 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T17:55:30.949 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T17:55:30.949 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T17:55:30.949 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T17:55:30.949 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T17:55:30.949 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T17:55:30.949 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T17:55:30.949 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T17:55:30.949 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T17:55:30.949 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T17:55:30.949 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T17:55:30.949 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T17:55:30.950 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T17:55:30.950 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T17:55:30.950 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T17:55:30.950 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T17:55:30.950 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T17:55:30.950 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T17:55:30.950 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T17:55:30.950 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T17:55:30.950 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T17:55:30.950 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T17:55:30.950 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T17:55:30.950 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T17:55:30.950 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T17:55:30.951 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T17:55:30.951 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T17:55:30.951 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T17:55:30.951 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T17:55:30.951 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T17:55:30.951 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T17:55:31.016 INFO:teuthology.orchestra.run.smithi018.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T17:55:31.019 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T17:55:31.066 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T17:55:31.094 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.18:6789]'), ('mon.c', '[v1:172.21.15.18:6790]'), ('mon.b', '[v1:172.21.15.169:6789]')] 2024-03-25T17:55:31.094 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.18:6789],[v1:172.21.15.18:6790],[v1:172.21.15.169:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T17:55:31.095 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:55:31.095 DEBUG:teuthology.orchestra.run.smithi018:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T17:55:31.149 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.18:6789]' --addv c '[v1:172.21.15.18:6790]' --addv b '[v1:172.21.15.169:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:55:31.236 INFO:teuthology.orchestra.run.smithi018.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:55:31.236 INFO:teuthology.orchestra.run.smithi018.stdout:monmaptool: generated fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:55:31.236 INFO:teuthology.orchestra.run.smithi018.stdout:setting min_mon_release = quincy 2024-03-25T17:55:31.236 INFO:teuthology.orchestra.run.smithi018.stdout:epoch 0 2024-03-25T17:55:31.236 INFO:teuthology.orchestra.run.smithi018.stdout:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:55:31.237 INFO:teuthology.orchestra.run.smithi018.stdout:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:55:31.237 INFO:teuthology.orchestra.run.smithi018.stdout:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:55:31.237 INFO:teuthology.orchestra.run.smithi018.stdout:min_mon_release 17 (quincy) 2024-03-25T17:55:31.237 INFO:teuthology.orchestra.run.smithi018.stdout:election_strategy: 3 2024-03-25T17:55:31.237 INFO:teuthology.orchestra.run.smithi018.stdout:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:55:31.237 INFO:teuthology.orchestra.run.smithi018.stdout:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:55:31.237 INFO:teuthology.orchestra.run.smithi018.stdout:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:55:31.237 INFO:teuthology.orchestra.run.smithi018.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T17:55:31.238 DEBUG:teuthology.orchestra.run.smithi018:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T17:55:31.252 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c6c98ae9-f2e3-42f0-a1fb-d587f9832636... 2024-03-25T17:55:31.253 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T17:55:31.295 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T17:55:31.342 INFO:teuthology.orchestra.run.smithi018.stdout:[global] 2024-03-25T17:55:31.342 INFO:teuthology.orchestra.run.smithi018.stdout: chdir = "" 2024-03-25T17:55:31.343 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T17:55:31.344 INFO:teuthology.orchestra.run.smithi018.stdout: auth supported = cephx 2024-03-25T17:55:31.344 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.345 INFO:teuthology.orchestra.run.smithi018.stdout: filestore xattr use omap = true 2024-03-25T17:55:31.345 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.345 INFO:teuthology.orchestra.run.smithi018.stdout: mon clock drift allowed = 1.000 2024-03-25T17:55:31.345 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.345 INFO:teuthology.orchestra.run.smithi018.stdout: osd crush chooseleaf type = 0 2024-03-25T17:55:31.346 INFO:teuthology.orchestra.run.smithi018.stdout: auth debug = true 2024-03-25T17:55:31.346 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.347 INFO:teuthology.orchestra.run.smithi018.stdout: ms die on old message = true 2024-03-25T17:55:31.347 INFO:teuthology.orchestra.run.smithi018.stdout: ms die on bug = true 2024-03-25T17:55:31.347 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.347 INFO:teuthology.orchestra.run.smithi018.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T17:55:31.347 INFO:teuthology.orchestra.run.smithi018.stdout: mon pg warn max object skew = 0 2024-03-25T17:55:31.347 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.348 INFO:teuthology.orchestra.run.smithi018.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T17:55:31.348 INFO:teuthology.orchestra.run.smithi018.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T17:55:31.349 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.349 INFO:teuthology.orchestra.run.smithi018.stdout: osd pool default size = 2 2024-03-25T17:55:31.349 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.349 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd allow primary affinity = true 2024-03-25T17:55:31.349 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd allow pg remap = true 2024-03-25T17:55:31.349 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on legacy crush tunables = false 2024-03-25T17:55:31.350 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on crush straw calc version zero = false 2024-03-25T17:55:31.350 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on no sortbitwise = false 2024-03-25T17:55:31.350 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on osd down out interval zero = false 2024-03-25T17:55:31.350 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on too few osds = false 2024-03-25T17:55:31.350 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T17:55:31.350 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T17:55:31.351 INFO:teuthology.orchestra.run.smithi018.stdout: mon_allow_pool_size_one = true 2024-03-25T17:55:31.351 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.351 INFO:teuthology.orchestra.run.smithi018.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T17:55:31.351 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.351 INFO:teuthology.orchestra.run.smithi018.stdout: osd default data pool replay window = 5 2024-03-25T17:55:31.352 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.352 INFO:teuthology.orchestra.run.smithi018.stdout: mon allow pool delete = true 2024-03-25T17:55:31.352 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.352 INFO:teuthology.orchestra.run.smithi018.stdout: mon cluster log file level = debug 2024-03-25T17:55:31.352 INFO:teuthology.orchestra.run.smithi018.stdout: debug asserts on shutdown = true 2024-03-25T17:55:31.352 INFO:teuthology.orchestra.run.smithi018.stdout: mon health detail to clog = false 2024-03-25T17:55:31.353 INFO:teuthology.orchestra.run.smithi018.stdout: mon host = "[v1:172.21.15.18:6789],[v1:172.21.15.18:6790],[v1:172.21.15.169:6789]" 2024-03-25T17:55:31.353 INFO:teuthology.orchestra.run.smithi018.stdout: mon client directed command retry = 5 2024-03-25T17:55:31.353 INFO:teuthology.orchestra.run.smithi018.stdout: mon election default strategy = 3 2024-03-25T17:55:31.353 INFO:teuthology.orchestra.run.smithi018.stdout: ms bind msgr2 = False 2024-03-25T17:55:31.353 INFO:teuthology.orchestra.run.smithi018.stdout: ms inject socket failures = 5000 2024-03-25T17:55:31.353 INFO:teuthology.orchestra.run.smithi018.stdout: ms type = async 2024-03-25T17:55:31.354 INFO:teuthology.orchestra.run.smithi018.stdout: osd heartbeat grace = 80 2024-03-25T17:55:31.354 INFO:teuthology.orchestra.run.smithi018.stdout: fsid = c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:55:31.354 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.354 INFO:teuthology.orchestra.run.smithi018.stdout:[osd] 2024-03-25T17:55:31.354 INFO:teuthology.orchestra.run.smithi018.stdout: osd journal size = 100 2024-03-25T17:55:31.355 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.355 INFO:teuthology.orchestra.run.smithi018.stdout: osd scrub load threshold = 5.0 2024-03-25T17:55:31.355 INFO:teuthology.orchestra.run.smithi018.stdout: osd scrub max interval = 600 2024-03-25T17:55:31.355 INFO:teuthology.orchestra.run.smithi018.stdout: osd mclock profile = high_recovery_ops 2024-03-25T17:55:31.355 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.356 INFO:teuthology.orchestra.run.smithi018.stdout: osd recover clone overlap = true 2024-03-25T17:55:31.356 INFO:teuthology.orchestra.run.smithi018.stdout: osd recovery max chunk = 1048576 2024-03-25T17:55:31.356 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.356 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug shutdown = true 2024-03-25T17:55:31.356 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug op order = true 2024-03-25T17:55:31.357 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug verify stray on activate = true 2024-03-25T17:55:31.357 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug trim objects = true 2024-03-25T17:55:31.357 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.357 INFO:teuthology.orchestra.run.smithi018.stdout: osd open classes on start = true 2024-03-25T17:55:31.358 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug pg log writeout = true 2024-03-25T17:55:31.358 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.358 INFO:teuthology.orchestra.run.smithi018.stdout: osd deep scrub update digest min age = 30 2024-03-25T17:55:31.358 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.358 INFO:teuthology.orchestra.run.smithi018.stdout: osd map max advance = 10 2024-03-25T17:55:31.359 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.359 INFO:teuthology.orchestra.run.smithi018.stdout: journal zero on create = true 2024-03-25T17:55:31.359 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.359 INFO:teuthology.orchestra.run.smithi018.stdout: filestore ondisk finisher threads = 3 2024-03-25T17:55:31.359 INFO:teuthology.orchestra.run.smithi018.stdout: filestore apply finisher threads = 3 2024-03-25T17:55:31.359 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.360 INFO:teuthology.orchestra.run.smithi018.stdout: bdev debug aio = true 2024-03-25T17:55:31.360 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug misdirected ops = true 2024-03-25T17:55:31.360 INFO:teuthology.orchestra.run.smithi018.stdout: bdev async discard = True 2024-03-25T17:55:31.360 INFO:teuthology.orchestra.run.smithi018.stdout: bdev enable discard = True 2024-03-25T17:55:31.360 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore allocator = stupid 2024-03-25T17:55:31.361 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore block size = 96636764160 2024-03-25T17:55:31.361 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore fsck on mount = True 2024-03-25T17:55:31.361 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore zero block detection = True 2024-03-25T17:55:31.361 INFO:teuthology.orchestra.run.smithi018.stdout: debug bluefs = 1 2024-03-25T17:55:31.361 INFO:teuthology.orchestra.run.smithi018.stdout: debug bluestore = 1 2024-03-25T17:55:31.361 INFO:teuthology.orchestra.run.smithi018.stdout: debug monc = 20 2024-03-25T17:55:31.362 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-03-25T17:55:31.362 INFO:teuthology.orchestra.run.smithi018.stdout: debug osd = 20 2024-03-25T17:55:31.362 INFO:teuthology.orchestra.run.smithi018.stdout: debug rocksdb = 10 2024-03-25T17:55:31.362 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T17:55:31.362 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd full ratio = 0.9 2024-03-25T17:55:31.363 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd nearfull ratio = 0.8 2024-03-25T17:55:31.363 INFO:teuthology.orchestra.run.smithi018.stdout: osd blocked scrub grace period = 3600 2024-03-25T17:55:31.363 INFO:teuthology.orchestra.run.smithi018.stdout: osd class default list = * 2024-03-25T17:55:31.363 INFO:teuthology.orchestra.run.smithi018.stdout: osd class load list = * 2024-03-25T17:55:31.363 INFO:teuthology.orchestra.run.smithi018.stdout: osd client watch timeout = 120 2024-03-25T17:55:31.363 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug verify cached snaps = True 2024-03-25T17:55:31.364 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug verify missing on start = True 2024-03-25T17:55:31.364 INFO:teuthology.orchestra.run.smithi018.stdout: osd failsafe full ratio = 0.95 2024-03-25T17:55:31.364 INFO:teuthology.orchestra.run.smithi018.stdout: osd fast shutdown = False 2024-03-25T17:55:31.364 INFO:teuthology.orchestra.run.smithi018.stdout: osd max markdown count = 1000 2024-03-25T17:55:31.364 INFO:teuthology.orchestra.run.smithi018.stdout: osd mclock override recovery settings = True 2024-03-25T17:55:31.364 INFO:teuthology.orchestra.run.smithi018.stdout: osd objectstore = bluestore 2024-03-25T17:55:31.365 INFO:teuthology.orchestra.run.smithi018.stdout: osd op queue = debug_random 2024-03-25T17:55:31.365 INFO:teuthology.orchestra.run.smithi018.stdout: osd op queue cut off = debug_random 2024-03-25T17:55:31.365 INFO:teuthology.orchestra.run.smithi018.stdout: osd shutdown pgref assert = True 2024-03-25T17:55:31.365 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.366 INFO:teuthology.orchestra.run.smithi018.stdout:[mgr] 2024-03-25T17:55:31.366 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-03-25T17:55:31.366 INFO:teuthology.orchestra.run.smithi018.stdout: debug mgr = 20 2024-03-25T17:55:31.366 INFO:teuthology.orchestra.run.smithi018.stdout: debug mon = 20 2024-03-25T17:55:31.366 INFO:teuthology.orchestra.run.smithi018.stdout: debug auth = 20 2024-03-25T17:55:31.367 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min pgs per osd = 4 2024-03-25T17:55:31.367 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min bytes per osd = 10 2024-03-25T17:55:31.367 INFO:teuthology.orchestra.run.smithi018.stdout: mgr/telemetry/nag = false 2024-03-25T17:55:31.367 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.367 INFO:teuthology.orchestra.run.smithi018.stdout:[mon] 2024-03-25T17:55:31.367 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-03-25T17:55:31.368 INFO:teuthology.orchestra.run.smithi018.stdout: debug mon = 20 2024-03-25T17:55:31.368 INFO:teuthology.orchestra.run.smithi018.stdout: debug paxos = 20 2024-03-25T17:55:31.368 INFO:teuthology.orchestra.run.smithi018.stdout: debug auth = 20 2024-03-25T17:55:31.368 INFO:teuthology.orchestra.run.smithi018.stdout: mon data avail warn = 5 2024-03-25T17:55:31.368 INFO:teuthology.orchestra.run.smithi018.stdout: mon mgr mkfs grace = 240 2024-03-25T17:55:31.368 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min pgs per osd = 4 2024-03-25T17:55:31.369 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd reporter subtree level = osd 2024-03-25T17:55:31.369 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd prime pg temp = true 2024-03-25T17:55:31.369 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min bytes per osd = 10 2024-03-25T17:55:31.369 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.369 INFO:teuthology.orchestra.run.smithi018.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T17:55:31.369 INFO:teuthology.orchestra.run.smithi018.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T17:55:31.370 INFO:teuthology.orchestra.run.smithi018.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T17:55:31.370 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.370 INFO:teuthology.orchestra.run.smithi018.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T17:55:31.370 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T17:55:31.370 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T17:55:31.370 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.371 INFO:teuthology.orchestra.run.smithi018.stdout: # 1m isn't quite enough 2024-03-25T17:55:31.371 INFO:teuthology.orchestra.run.smithi018.stdout: mon_down_mkfs_grace = 2m 2024-03-25T17:55:31.371 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.371 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_filestore_osds = false 2024-03-25T17:55:31.371 INFO:teuthology.orchestra.run.smithi018.stdout: mon min osdmap epochs = 50 2024-03-25T17:55:31.371 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd crush smoke test = False 2024-03-25T17:55:31.372 INFO:teuthology.orchestra.run.smithi018.stdout: mon osdmap full prune interval = 2 2024-03-25T17:55:31.372 INFO:teuthology.orchestra.run.smithi018.stdout: mon osdmap full prune min = 15 2024-03-25T17:55:31.372 INFO:teuthology.orchestra.run.smithi018.stdout: mon osdmap full prune txsize = 2 2024-03-25T17:55:31.372 INFO:teuthology.orchestra.run.smithi018.stdout: mon scrub interval = 300 2024-03-25T17:55:31.372 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on pool no app = False 2024-03-25T17:55:31.372 INFO:teuthology.orchestra.run.smithi018.stdout: paxos service trim min = 10 2024-03-25T17:55:31.373 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:55:31.373 INFO:teuthology.orchestra.run.smithi018.stdout:[client] 2024-03-25T17:55:31.373 INFO:teuthology.orchestra.run.smithi018.stdout: rgw cache enabled = true 2024-03-25T17:55:31.373 INFO:teuthology.orchestra.run.smithi018.stdout: rgw enable ops log = true 2024-03-25T17:55:31.373 INFO:teuthology.orchestra.run.smithi018.stdout: rgw enable usage log = true 2024-03-25T17:55:31.373 INFO:teuthology.orchestra.run.smithi018.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T17:55:31.374 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T17:55:31.374 INFO:teuthology.orchestra.run.smithi018.stdout: debug monc = 20 2024-03-25T17:55:31.374 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-03-25T17:55:31.374 INFO:teuthology.orchestra.run.smithi018.stdout: debug objecter = 20 2024-03-25T17:55:31.374 INFO:teuthology.orchestra.run.smithi018.stdout: debug rados = 20 2024-03-25T17:55:31.375 INFO:teuthology.orchestra.run.smithi018.stdout:[mon.a] 2024-03-25T17:55:31.375 INFO:teuthology.orchestra.run.smithi018.stdout:[mon.c] 2024-03-25T17:55:31.375 INFO:teuthology.orchestra.run.smithi018.stdout:[mon.b] 2024-03-25T17:55:31.378 INFO:teuthology.orchestra.run.smithi169.stdout:[global] 2024-03-25T17:55:31.378 INFO:teuthology.orchestra.run.smithi169.stdout: chdir = "" 2024-03-25T17:55:31.378 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T17:55:31.378 INFO:teuthology.orchestra.run.smithi169.stdout: auth supported = cephx 2024-03-25T17:55:31.378 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.379 INFO:teuthology.orchestra.run.smithi169.stdout: filestore xattr use omap = true 2024-03-25T17:55:31.379 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.379 INFO:teuthology.orchestra.run.smithi169.stdout: mon clock drift allowed = 1.000 2024-03-25T17:55:31.379 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.379 INFO:teuthology.orchestra.run.smithi169.stdout: osd crush chooseleaf type = 0 2024-03-25T17:55:31.379 INFO:teuthology.orchestra.run.smithi169.stdout: auth debug = true 2024-03-25T17:55:31.380 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.380 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on old message = true 2024-03-25T17:55:31.380 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on bug = true 2024-03-25T17:55:31.380 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.380 INFO:teuthology.orchestra.run.smithi169.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T17:55:31.380 INFO:teuthology.orchestra.run.smithi169.stdout: mon pg warn max object skew = 0 2024-03-25T17:55:31.381 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.381 INFO:teuthology.orchestra.run.smithi169.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T17:55:31.381 INFO:teuthology.orchestra.run.smithi169.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T17:55:31.381 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.381 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default size = 2 2024-03-25T17:55:31.381 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.382 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow primary affinity = true 2024-03-25T17:55:31.382 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow pg remap = true 2024-03-25T17:55:31.382 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on legacy crush tunables = false 2024-03-25T17:55:31.382 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on crush straw calc version zero = false 2024-03-25T17:55:31.382 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on no sortbitwise = false 2024-03-25T17:55:31.382 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on osd down out interval zero = false 2024-03-25T17:55:31.383 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on too few osds = false 2024-03-25T17:55:31.383 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T17:55:31.383 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T17:55:31.383 INFO:teuthology.orchestra.run.smithi169.stdout: mon_allow_pool_size_one = true 2024-03-25T17:55:31.383 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.383 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T17:55:31.384 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.384 INFO:teuthology.orchestra.run.smithi169.stdout: osd default data pool replay window = 5 2024-03-25T17:55:31.384 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.384 INFO:teuthology.orchestra.run.smithi169.stdout: mon allow pool delete = true 2024-03-25T17:55:31.384 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.385 INFO:teuthology.orchestra.run.smithi169.stdout: mon cluster log file level = debug 2024-03-25T17:55:31.385 INFO:teuthology.orchestra.run.smithi169.stdout: debug asserts on shutdown = true 2024-03-25T17:55:31.385 INFO:teuthology.orchestra.run.smithi169.stdout: mon health detail to clog = false 2024-03-25T17:55:31.385 INFO:teuthology.orchestra.run.smithi169.stdout: mon host = "[v1:172.21.15.18:6789],[v1:172.21.15.18:6790],[v1:172.21.15.169:6789]" 2024-03-25T17:55:31.385 INFO:teuthology.orchestra.run.smithi169.stdout: mon client directed command retry = 5 2024-03-25T17:55:31.385 INFO:teuthology.orchestra.run.smithi169.stdout: mon election default strategy = 3 2024-03-25T17:55:31.386 INFO:teuthology.orchestra.run.smithi169.stdout: ms bind msgr2 = False 2024-03-25T17:55:31.386 INFO:teuthology.orchestra.run.smithi169.stdout: ms inject socket failures = 5000 2024-03-25T17:55:31.386 INFO:teuthology.orchestra.run.smithi169.stdout: ms type = async 2024-03-25T17:55:31.386 INFO:teuthology.orchestra.run.smithi169.stdout: osd heartbeat grace = 80 2024-03-25T17:55:31.386 INFO:teuthology.orchestra.run.smithi169.stdout: fsid = c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:55:31.387 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.387 INFO:teuthology.orchestra.run.smithi169.stdout:[osd] 2024-03-25T17:55:31.387 INFO:teuthology.orchestra.run.smithi169.stdout: osd journal size = 100 2024-03-25T17:55:31.387 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.387 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub load threshold = 5.0 2024-03-25T17:55:31.387 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub max interval = 600 2024-03-25T17:55:31.388 INFO:teuthology.orchestra.run.smithi169.stdout: osd mclock profile = high_recovery_ops 2024-03-25T17:55:31.388 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.388 INFO:teuthology.orchestra.run.smithi169.stdout: osd recover clone overlap = true 2024-03-25T17:55:31.388 INFO:teuthology.orchestra.run.smithi169.stdout: osd recovery max chunk = 1048576 2024-03-25T17:55:31.388 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.388 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug shutdown = true 2024-03-25T17:55:31.389 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug op order = true 2024-03-25T17:55:31.389 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify stray on activate = true 2024-03-25T17:55:31.389 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug trim objects = true 2024-03-25T17:55:31.389 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.389 INFO:teuthology.orchestra.run.smithi169.stdout: osd open classes on start = true 2024-03-25T17:55:31.389 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug pg log writeout = true 2024-03-25T17:55:31.390 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.390 INFO:teuthology.orchestra.run.smithi169.stdout: osd deep scrub update digest min age = 30 2024-03-25T17:55:31.390 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.390 INFO:teuthology.orchestra.run.smithi169.stdout: osd map max advance = 10 2024-03-25T17:55:31.390 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.391 INFO:teuthology.orchestra.run.smithi169.stdout: journal zero on create = true 2024-03-25T17:55:31.391 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.391 INFO:teuthology.orchestra.run.smithi169.stdout: filestore ondisk finisher threads = 3 2024-03-25T17:55:31.391 INFO:teuthology.orchestra.run.smithi169.stdout: filestore apply finisher threads = 3 2024-03-25T17:55:31.391 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.391 INFO:teuthology.orchestra.run.smithi169.stdout: bdev debug aio = true 2024-03-25T17:55:31.392 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug misdirected ops = true 2024-03-25T17:55:31.392 INFO:teuthology.orchestra.run.smithi169.stdout: bdev async discard = True 2024-03-25T17:55:31.392 INFO:teuthology.orchestra.run.smithi169.stdout: bdev enable discard = True 2024-03-25T17:55:31.392 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore allocator = stupid 2024-03-25T17:55:31.392 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore block size = 96636764160 2024-03-25T17:55:31.392 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore fsck on mount = True 2024-03-25T17:55:31.393 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore zero block detection = True 2024-03-25T17:55:31.393 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluefs = 1 2024-03-25T17:55:31.393 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluestore = 1 2024-03-25T17:55:31.393 INFO:teuthology.orchestra.run.smithi169.stdout: debug monc = 20 2024-03-25T17:55:31.393 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-25T17:55:31.393 INFO:teuthology.orchestra.run.smithi169.stdout: debug osd = 20 2024-03-25T17:55:31.393 INFO:teuthology.orchestra.run.smithi169.stdout: debug rocksdb = 10 2024-03-25T17:55:31.394 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T17:55:31.394 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd full ratio = 0.9 2024-03-25T17:55:31.394 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd nearfull ratio = 0.8 2024-03-25T17:55:31.394 INFO:teuthology.orchestra.run.smithi169.stdout: osd blocked scrub grace period = 3600 2024-03-25T17:55:31.394 INFO:teuthology.orchestra.run.smithi169.stdout: osd class default list = * 2024-03-25T17:55:31.395 INFO:teuthology.orchestra.run.smithi169.stdout: osd class load list = * 2024-03-25T17:55:31.395 INFO:teuthology.orchestra.run.smithi169.stdout: osd client watch timeout = 120 2024-03-25T17:55:31.395 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify cached snaps = True 2024-03-25T17:55:31.395 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify missing on start = True 2024-03-25T17:55:31.395 INFO:teuthology.orchestra.run.smithi169.stdout: osd failsafe full ratio = 0.95 2024-03-25T17:55:31.395 INFO:teuthology.orchestra.run.smithi169.stdout: osd fast shutdown = False 2024-03-25T17:55:31.395 INFO:teuthology.orchestra.run.smithi169.stdout: osd max markdown count = 1000 2024-03-25T17:55:31.395 INFO:teuthology.orchestra.run.smithi169.stdout: osd mclock override recovery settings = True 2024-03-25T17:55:31.395 INFO:teuthology.orchestra.run.smithi169.stdout: osd objectstore = bluestore 2024-03-25T17:55:31.396 INFO:teuthology.orchestra.run.smithi169.stdout: osd op queue = debug_random 2024-03-25T17:55:31.396 INFO:teuthology.orchestra.run.smithi169.stdout: osd op queue cut off = debug_random 2024-03-25T17:55:31.396 INFO:teuthology.orchestra.run.smithi169.stdout: osd shutdown pgref assert = True 2024-03-25T17:55:31.396 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.396 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr] 2024-03-25T17:55:31.396 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-25T17:55:31.396 INFO:teuthology.orchestra.run.smithi169.stdout: debug mgr = 20 2024-03-25T17:55:31.396 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-25T17:55:31.396 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-25T17:55:31.397 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-25T17:55:31.397 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-25T17:55:31.397 INFO:teuthology.orchestra.run.smithi169.stdout: mgr/telemetry/nag = false 2024-03-25T17:55:31.397 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.397 INFO:teuthology.orchestra.run.smithi169.stdout:[mon] 2024-03-25T17:55:31.397 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-25T17:55:31.397 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-03-25T17:55:31.397 INFO:teuthology.orchestra.run.smithi169.stdout: debug paxos = 20 2024-03-25T17:55:31.397 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-03-25T17:55:31.397 INFO:teuthology.orchestra.run.smithi169.stdout: mon data avail warn = 5 2024-03-25T17:55:31.397 INFO:teuthology.orchestra.run.smithi169.stdout: mon mgr mkfs grace = 240 2024-03-25T17:55:31.397 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-03-25T17:55:31.397 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd reporter subtree level = osd 2024-03-25T17:55:31.398 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd prime pg temp = true 2024-03-25T17:55:31.398 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-03-25T17:55:31.398 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.398 INFO:teuthology.orchestra.run.smithi169.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T17:55:31.398 INFO:teuthology.orchestra.run.smithi169.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T17:55:31.398 INFO:teuthology.orchestra.run.smithi169.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T17:55:31.398 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.398 INFO:teuthology.orchestra.run.smithi169.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T17:55:31.398 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T17:55:31.398 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T17:55:31.398 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.398 INFO:teuthology.orchestra.run.smithi169.stdout: # 1m isn't quite enough 2024-03-25T17:55:31.398 INFO:teuthology.orchestra.run.smithi169.stdout: mon_down_mkfs_grace = 2m 2024-03-25T17:55:31.399 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.399 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_filestore_osds = false 2024-03-25T17:55:31.399 INFO:teuthology.orchestra.run.smithi169.stdout: mon min osdmap epochs = 50 2024-03-25T17:55:31.399 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd crush smoke test = False 2024-03-25T17:55:31.399 INFO:teuthology.orchestra.run.smithi169.stdout: mon osdmap full prune interval = 2 2024-03-25T17:55:31.399 INFO:teuthology.orchestra.run.smithi169.stdout: mon osdmap full prune min = 15 2024-03-25T17:55:31.399 INFO:teuthology.orchestra.run.smithi169.stdout: mon osdmap full prune txsize = 2 2024-03-25T17:55:31.399 INFO:teuthology.orchestra.run.smithi169.stdout: mon scrub interval = 300 2024-03-25T17:55:31.399 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on pool no app = False 2024-03-25T17:55:31.399 INFO:teuthology.orchestra.run.smithi169.stdout: paxos service trim min = 10 2024-03-25T17:55:31.399 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-03-25T17:55:31.399 INFO:teuthology.orchestra.run.smithi169.stdout:[client] 2024-03-25T17:55:31.399 INFO:teuthology.orchestra.run.smithi169.stdout: rgw cache enabled = true 2024-03-25T17:55:31.400 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable ops log = true 2024-03-25T17:55:31.400 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable usage log = true 2024-03-25T17:55:31.400 INFO:teuthology.orchestra.run.smithi169.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T17:55:31.400 INFO:teuthology.orchestra.run.smithi169.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T17:55:31.400 INFO:teuthology.orchestra.run.smithi169.stdout: debug monc = 20 2024-03-25T17:55:31.400 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-03-25T17:55:31.400 INFO:teuthology.orchestra.run.smithi169.stdout: debug objecter = 20 2024-03-25T17:55:31.400 INFO:teuthology.orchestra.run.smithi169.stdout: debug rados = 20 2024-03-25T17:55:31.400 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.a] 2024-03-25T17:55:31.401 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.c] 2024-03-25T17:55:31.401 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.b] 2024-03-25T17:55:31.401 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T17:55:31.401 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T17:55:31.456 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T17:55:31.456 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:55:31.456 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T17:55:31.470 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:55:31.470 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T17:55:31.525 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T17:55:31.525 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:55:31.525 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T17:55:31.525 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T17:55:31.605 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:55:31.605 DEBUG:teuthology.orchestra.run.smithi018:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:55:31.658 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T17:55:31.659 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T17:55:31.659 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T17:55:31.659 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T17:55:31.699 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T17:55:31.699 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:55:31.754 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T17:55:31.754 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T17:55:31.754 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T17:55:31.808 INFO:teuthology.orchestra.run.smithi018.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T17:55:31.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T17:55:31.872 INFO:teuthology.orchestra.run.smithi169.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T17:55:31.874 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T17:55:31.875 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T17:55:31.875 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T17:55:31.916 INFO:teuthology.orchestra.run.smithi018.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T17:55:31.932 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T17:55:31.974 INFO:teuthology.orchestra.run.smithi169.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T17:55:31.989 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T17:55:31.989 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi018.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi169.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T17:55:31.990 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T17:55:32.016 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:55:32.016 INFO:tasks.ceph:role: osd.0 2024-03-25T17:55:32.016 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T17:55:32.016 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T17:55:32.082 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:55:32.083 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:55:32.083 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:55:32.083 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:55:32.083 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:55:32.083 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:55:32.083 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:55:32.084 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:55:32.084 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:55:32.084 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:55:32.116 INFO:teuthology.orchestra.run.smithi018.stdout:Discarding blocks...Done. 2024-03-25T17:55:32.118 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2024-03-25T17:55:32.118 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T17:55:32.189 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T17:55:32.265 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T17:55:32.332 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:55:32.332 INFO:tasks.ceph:role: osd.1 2024-03-25T17:55:32.332 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T17:55:32.332 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T17:55:32.398 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:55:32.398 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:55:32.399 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:55:32.399 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:55:32.399 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:55:32.399 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:55:32.399 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:55:32.399 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:55:32.399 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:55:32.399 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:55:32.430 INFO:teuthology.orchestra.run.smithi018.stdout:Discarding blocks...Done. 2024-03-25T17:55:32.432 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2024-03-25T17:55:32.433 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T17:55:32.507 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T17:55:32.580 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T17:55:32.647 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:55:32.647 INFO:tasks.ceph:role: osd.2 2024-03-25T17:55:32.647 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T17:55:32.647 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T17:55:32.713 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:55:32.713 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:55:32.713 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:55:32.713 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:55:32.713 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:55:32.713 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:55:32.714 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:55:32.714 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:55:32.714 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:55:32.714 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:55:32.745 INFO:teuthology.orchestra.run.smithi018.stdout:Discarding blocks...Done. 2024-03-25T17:55:32.747 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2024-03-25T17:55:32.747 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T17:55:32.817 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T17:55:32.892 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T17:55:32.959 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:55:32.959 INFO:tasks.ceph:role: osd.3 2024-03-25T17:55:32.959 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T17:55:32.960 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T17:55:33.025 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:55:33.025 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:55:33.025 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:55:33.025 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:55:33.025 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:55:33.025 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:55:33.025 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:55:33.026 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:55:33.026 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:55:33.026 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:55:33.057 INFO:teuthology.orchestra.run.smithi018.stdout:Discarding blocks...Done. 2024-03-25T17:55:33.058 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2024-03-25T17:55:33.059 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T17:55:33.128 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T17:55:33.199 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:55:33.292 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:33.292+0000 7fcecf615640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T17:55:33.292 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:33.292+0000 7fcecf615640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T17:55:33.292 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:33.292+0000 7fcecf615640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T17:55:33.292 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:33.292+0000 7fcecf615640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T17:55:33.292 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:33.292+0000 7fcecf615640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T17:55:33.292 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:33.292+0000 7fcecf615640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T17:55:36.126 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T17:55:36.157 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:55:36.248 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:36.248+0000 7f2ba5e1c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T17:55:36.248 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:36.248+0000 7f2ba5e1c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T17:55:36.248 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:36.248+0000 7f2ba5e1c640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T17:55:36.248 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:36.248+0000 7f2ba5e1c640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T17:55:36.248 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:36.248+0000 7f2ba5e1c640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T17:55:36.248 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:36.248+0000 7f2ba5e1c640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T17:55:39.094 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T17:55:39.125 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:55:39.217 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:39.217+0000 7f484cc12640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T17:55:39.217 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:39.217+0000 7f484cc12640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T17:55:39.217 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:39.217+0000 7f484cc12640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T17:55:39.217 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:39.217+0000 7f484cc12640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T17:55:39.217 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:39.217+0000 7f484cc12640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T17:55:39.217 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:39.217+0000 7f484cc12640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T17:55:42.053 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T17:55:42.084 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:55:42.176 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:42.175+0000 7f8de249d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T17:55:42.176 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:42.175+0000 7f8de249d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T17:55:42.176 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:42.175+0000 7f8de249d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T17:55:42.177 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:42.175+0000 7f8de249d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T17:55:42.177 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:42.175+0000 7f8de249d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T17:55:42.177 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:55:42.176+0000 7f8de249d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T17:55:45.017 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T17:55:45.048 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T17:55:45.080 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:55:45.080 INFO:tasks.ceph:role: osd.4 2024-03-25T17:55:45.080 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T17:55:45.080 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T17:55:45.158 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:55:45.158 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:55:45.158 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:55:45.158 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:55:45.158 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:55:45.158 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:55:45.158 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:55:45.158 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:55:45.158 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:55:45.158 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:55:45.679 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-03-25T17:55:45.681 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-03-25T17:55:45.682 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T17:55:45.838 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T17:55:45.874 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T17:55:45.943 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:55:45.944 INFO:tasks.ceph:role: osd.5 2024-03-25T17:55:45.944 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T17:55:45.944 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T17:55:46.019 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:55:46.019 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:55:46.020 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:55:46.020 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:55:46.020 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:55:46.020 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:55:46.020 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:55:46.020 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:55:46.020 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:55:46.020 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:55:46.467 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-03-25T17:55:46.469 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-03-25T17:55:46.469 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T17:55:46.558 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T17:55:46.593 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T17:55:46.660 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:55:46.661 INFO:tasks.ceph:role: osd.6 2024-03-25T17:55:46.661 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T17:55:46.661 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T17:55:46.736 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:55:46.737 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:55:46.737 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:55:46.737 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:55:46.737 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:55:46.737 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:55:46.738 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:55:46.738 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:55:46.738 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:55:46.738 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:55:47.229 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-03-25T17:55:47.231 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-03-25T17:55:47.232 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T17:55:47.359 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T17:55:47.395 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T17:55:47.462 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:55:47.462 INFO:tasks.ceph:role: osd.7 2024-03-25T17:55:47.463 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T17:55:47.463 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T17:55:47.537 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:55:47.537 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:55:47.537 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:55:47.537 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:55:47.537 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:55:47.537 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:55:47.537 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:55:47.537 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:55:47.537 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:55:47.537 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:55:48.038 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-03-25T17:55:48.039 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-03-25T17:55:48.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T17:55:48.167 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T17:55:48.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:55:48.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:48.294+0000 7f5cc400e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T17:55:48.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:48.295+0000 7f5cc400e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T17:55:48.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:48.295+0000 7f5cc400e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T17:55:48.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:48.295+0000 7f5cc400e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T17:55:48.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:48.295+0000 7f5cc400e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T17:55:48.295 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:48.295+0000 7f5cc400e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T17:55:51.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T17:55:51.175 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:55:51.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:51.266+0000 7faafca0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T17:55:51.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:51.266+0000 7faafca0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T17:55:51.267 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:51.266+0000 7faafca0c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T17:55:51.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:51.266+0000 7faafca0c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T17:55:51.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:51.266+0000 7faafca0c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T17:55:51.268 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:51.267+0000 7faafca0c640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T17:55:54.111 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T17:55:54.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:55:54.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:54.234+0000 7fdf3a9f9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T17:55:54.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:54.234+0000 7fdf3a9f9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T17:55:54.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:54.234+0000 7fdf3a9f9640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T17:55:54.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:54.234+0000 7fdf3a9f9640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T17:55:54.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:54.234+0000 7fdf3a9f9640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T17:55:54.235 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:54.235+0000 7fdf3a9f9640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T17:55:57.085 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T17:55:57.117 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:55:57.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:57.210+0000 7f118a014640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T17:55:57.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:57.211+0000 7f118a014640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T17:55:57.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:57.211+0000 7f118a014640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T17:55:57.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:57.211+0000 7f118a014640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T17:55:57.211 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:57.211+0000 7f118a014640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T17:55:57.212 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:55:57.211+0000 7f118a014640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T17:56:00.062 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T17:56:00.095 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T17:56:00.095 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:56:00.095 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T17:56:00.127 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:56:00.127 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T17:56:00.198 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:56:00.198 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T17:56:00.268 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:56:00.268 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T17:56:00.337 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:56:00.337 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T17:56:00.406 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T17:56:00.406 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T17:56:00.437 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T17:56:00.437 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T17:56:00.507 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T17:56:00.507 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T17:56:00.577 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T17:56:00.577 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T17:56:00.646 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T17:56:00.646 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T17:56:00.716 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:56:00.716 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T17:56:00.734 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T17:56:00.734 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T17:56:00.772 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T17:56:00.772 DEBUG:teuthology.orchestra.run.smithi018:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T17:56:00.777 DEBUG:teuthology.orchestra.run.smithi169:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T17:56:00.816 INFO:teuthology.orchestra.run.smithi018.stdout:[mgr.y] 2024-03-25T17:56:00.816 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCTugFmBcA3MBAASnjeoK2DVLT1ljCgatpFhg== 2024-03-25T17:56:00.816 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.0] 2024-03-25T17:56:00.816 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCVugFmIqNxERAAeJ0dEEFfzHyBy51Ncq1vyA== 2024-03-25T17:56:00.816 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.1] 2024-03-25T17:56:00.817 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCYugFmqf7QDhAAJ6wvyE01zpmGh+Thon0Atg== 2024-03-25T17:56:00.817 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.2] 2024-03-25T17:56:00.817 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCbugFmcDz3DBAAxJ8mq0DajKTdpxQBp2z61Q== 2024-03-25T17:56:00.817 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.3] 2024-03-25T17:56:00.817 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCeugFm5pOKChAAAR2cUY1DQOcCRVopZW2xfQ== 2024-03-25T17:56:00.817 INFO:teuthology.orchestra.run.smithi018.stdout:[mgr.x] 2024-03-25T17:56:00.817 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCTugFmPBoLNBAAdCbFmHgpH1L+a80rdltXvw== 2024-03-25T17:56:00.817 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.4] 2024-03-25T17:56:00.817 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCkugFm7CGgERAALjx1fkn74xoYgHjxnZWSsA== 2024-03-25T17:56:00.817 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.5] 2024-03-25T17:56:00.817 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCnugFmIoz7DxAAP6lfTGfo+W67/ysyTsUUzA== 2024-03-25T17:56:00.818 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.6] 2024-03-25T17:56:00.818 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCqugFmAWEGDhAAOuD6gbAgZjTqfI5O2YD7Yg== 2024-03-25T17:56:00.818 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.7] 2024-03-25T17:56:00.818 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCtugFm9kSeDBAAFYMcyNmD1iFFNb6iNDRajg== 2024-03-25T17:56:00.818 INFO:teuthology.orchestra.run.smithi018.stdout:[client.0] 2024-03-25T17:56:00.818 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCTugFmFcqyNhAAyHxD58JSkEdTW5mkD3wZJg== 2024-03-25T17:56:00.818 INFO:teuthology.orchestra.run.smithi018.stdout:[client.1] 2024-03-25T17:56:00.818 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCTugFmrkIkOhAA+KYBE9s1+Q/K84yUsy9NIQ== 2024-03-25T17:56:00.839 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr.y] 2024-03-25T17:56:00.839 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCTugFmBcA3MBAASnjeoK2DVLT1ljCgatpFhg== 2024-03-25T17:56:00.839 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.0] 2024-03-25T17:56:00.839 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCVugFmIqNxERAAeJ0dEEFfzHyBy51Ncq1vyA== 2024-03-25T17:56:00.839 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.1] 2024-03-25T17:56:00.839 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCYugFmqf7QDhAAJ6wvyE01zpmGh+Thon0Atg== 2024-03-25T17:56:00.839 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.2] 2024-03-25T17:56:00.839 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCbugFmcDz3DBAAxJ8mq0DajKTdpxQBp2z61Q== 2024-03-25T17:56:00.839 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.3] 2024-03-25T17:56:00.840 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCeugFm5pOKChAAAR2cUY1DQOcCRVopZW2xfQ== 2024-03-25T17:56:00.840 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr.x] 2024-03-25T17:56:00.840 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCTugFmPBoLNBAAdCbFmHgpH1L+a80rdltXvw== 2024-03-25T17:56:00.840 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.4] 2024-03-25T17:56:00.840 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCkugFm7CGgERAALjx1fkn74xoYgHjxnZWSsA== 2024-03-25T17:56:00.840 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.5] 2024-03-25T17:56:00.840 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCnugFmIoz7DxAAP6lfTGfo+W67/ysyTsUUzA== 2024-03-25T17:56:00.840 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.6] 2024-03-25T17:56:00.840 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCqugFmAWEGDhAAOuD6gbAgZjTqfI5O2YD7Yg== 2024-03-25T17:56:00.840 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.7] 2024-03-25T17:56:00.840 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCtugFm9kSeDBAAFYMcyNmD1iFFNb6iNDRajg== 2024-03-25T17:56:00.840 INFO:teuthology.orchestra.run.smithi169.stdout:[client.0] 2024-03-25T17:56:00.840 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCTugFmFcqyNhAAyHxD58JSkEdTW5mkD3wZJg== 2024-03-25T17:56:00.841 INFO:teuthology.orchestra.run.smithi169.stdout:[client.1] 2024-03-25T17:56:00.841 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCTugFmrkIkOhAA+KYBE9s1+Q/K84yUsy9NIQ== 2024-03-25T17:56:00.842 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T17:56:00.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T17:56:00.936 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:56:00.939 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:56:00.991 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:56:00.994 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:56:01.045 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:56:01.048 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:56:01.099 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:56:01.102 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:56:01.153 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T17:56:01.155 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T17:56:01.207 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:56:01.210 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:56:01.262 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:56:01.264 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:56:01.316 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:56:01.318 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:56:01.370 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:56:01.372 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:56:01.426 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T17:56:01.428 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T17:56:01.483 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T17:56:01.485 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T17:56:01.540 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T17:56:01.540 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T17:56:01.566 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T17:56:01.663 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T17:56:01.689 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T17:56:01.755 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T17:56:01.851 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T17:56:01.878 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T17:56:01.905 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T17:56:02.002 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T17:56:02.028 DEBUG:teuthology.orchestra.run.smithi018:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:56:02.030 DEBUG:teuthology.orchestra.run.smithi169:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:56:02.083 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T17:56:02.084 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T17:56:02.084 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T17:56:02.084 DEBUG:teuthology.orchestra.run.smithi018:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T17:56:02.086 INFO:tasks.ceph.mon.a:Started 2024-03-25T17:56:02.086 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T17:56:02.087 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T17:56:02.087 DEBUG:teuthology.orchestra.run.smithi018:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T17:56:02.088 INFO:tasks.ceph.mon.c:Started 2024-03-25T17:56:02.089 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T17:56:02.089 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T17:56:02.089 DEBUG:teuthology.orchestra.run.smithi169:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T17:56:02.126 INFO:tasks.ceph.mon.b:Started 2024-03-25T17:56:02.127 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T17:56:02.127 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T17:56:02.127 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T17:56:02.129 INFO:tasks.ceph.mgr.y:Started 2024-03-25T17:56:02.129 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T17:56:02.130 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T17:56:02.131 INFO:tasks.ceph.mgr.x:Started 2024-03-25T17:56:02.131 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T17:56:02.132 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T17:56:02.234 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.232+0000 7f8362fcf640 10 monclient: get_monmap_and_config 2024-03-25T17:56:02.234 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.232+0000 7f8362fcf640 10 monclient: build_initial_monmap 2024-03-25T17:56:02.234 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.232+0000 7f8362fcf640 10 monclient: monmap: 2024-03-25T17:56:02.234 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 0 2024-03-25T17:56:02.234 INFO:teuthology.orchestra.run.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:02.234 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T17:56:02.234226+0000 2024-03-25T17:56:02.234 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T17:56:02.234226+0000 2024-03-25T17:56:02.234 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:02.234 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:02.235 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:02.235 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:56:02.235 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:56:02.235 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T17:56:02.235 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.233+0000 7f8362fcf640 1 Processor -- start 2024-03-25T17:56:02.235 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.233+0000 7f8362fcf640 1 -- start start 2024-03-25T17:56:02.235 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.233+0000 7f8362fcf640 10 monclient: init 2024-03-25T17:56:02.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.234+0000 7f8362fcf640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:02.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.234+0000 7f8362fcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:56:02.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.234+0000 7f8362fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f835c0a9920 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:02.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.234+0000 7f8362fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f835c0b45a0 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:02.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.234+0000 7f8362fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f835c146450 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:02.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.234+0000 7f8362fcf640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c06b100 con 0x7f835c146450 2024-03-25T17:56:02.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.234+0000 7f8362fcf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c0b2940 con 0x7f835c0a9920 2024-03-25T17:56:02.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.234+0000 7f8362fcf640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c13fec0 con 0x7f835c0b45a0 2024-03-25T17:56:02.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.234+0000 7f8362fcf640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:02.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.234+0000 7f8362fcf640 10 monclient(hunting): authenticate will time out at 6890.812926s 2024-03-25T17:56:02.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.234+0000 7f835bfff640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f835c145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T17:56:02.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.234+0000 7f836156c640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f835c14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T17:56:02.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.234+0000 7f8360d6b640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 legacy=0x7f835c13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T17:56:02.435 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.435+0000 7f835bfff640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f835c145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T17:56:02.435 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.435+0000 7f836156c640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f835c14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T17:56:02.436 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.435+0000 7f8360d6b640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 legacy=0x7f835c13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T17:56:02.836 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.836+0000 7f835bfff640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f835c145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T17:56:02.836 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.836+0000 7f836156c640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f835c14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T17:56:02.837 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:02.836+0000 7f8360d6b640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 legacy=0x7f835c13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T17:56:03.637 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:03.637+0000 7f835bfff640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f835c145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T17:56:03.638 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:03.637+0000 7f836156c640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f835c14dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T17:56:03.638 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:03.637+0000 7f8360d6b640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 legacy=0x7f835c13f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T17:56:05.235 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f835a7fc640 10 monclient(hunting): tick 2024-03-25T17:56:05.235 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f835a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:56:05.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f835a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T17:56:05.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f835a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:56:05.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f835a7fc640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f835c145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:56:05.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f835a7fc640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 legacy=0x7f835c13f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:56:05.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f835a7fc640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f835c14dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:56:05.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f835a7fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:56:05.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f835c146450 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:05.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f835c0b45a0 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:05.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f835c0a9920 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:05.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f835a7fc640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c06b100 con 0x7f835c146450 2024-03-25T17:56:05.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f835a7fc640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c0b2940 con 0x7f835c0a9920 2024-03-25T17:56:05.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f8360d6b640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f83400056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T17:56:05.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f835a7fc640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c13fec0 con 0x7f835c0b45a0 2024-03-25T17:56:05.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f835a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:05.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f835bfff640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f8340008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T17:56:05.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.234+0000 7f836156c640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 legacy=0x7f834000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T17:56:05.435 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.435+0000 7f835bfff640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f8340008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T17:56:05.436 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.435+0000 7f8360d6b640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f83400056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T17:56:05.436 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.435+0000 7f836156c640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 legacy=0x7f834000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T17:56:05.836 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.836+0000 7f835bfff640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f8340008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T17:56:05.837 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.836+0000 7f8360d6b640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f83400056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T17:56:05.837 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:05.836+0000 7f836156c640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 legacy=0x7f834000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T17:56:06.638 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:06.637+0000 7f835bfff640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f8340008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T17:56:06.638 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:06.637+0000 7f8360d6b640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f83400056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T17:56:06.638 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:06.637+0000 7f836156c640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 legacy=0x7f834000c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T17:56:08.235 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f835a7fc640 10 monclient(hunting): tick 2024-03-25T17:56:08.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f835a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:56:08.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f835a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T17:56:08.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f835a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:56:08.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f835a7fc640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f8340008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:56:08.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f835a7fc640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 legacy=0x7f834000c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:56:08.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f835a7fc640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f83400056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:56:08.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f835a7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:56:08.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f835c146450 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:08.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f835c0a9920 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:08.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f835c0b45a0 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:08.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f835a7fc640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340010e20 con 0x7f835c146450 2024-03-25T17:56:08.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f835a7fc640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c06b100 con 0x7f835c0a9920 2024-03-25T17:56:08.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f835a7fc640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c0b2940 con 0x7f835c0b45a0 2024-03-25T17:56:08.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f835a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:08.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f8360d6b640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f8340005e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T17:56:08.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f836156c640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f8340008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T17:56:08.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.234+0000 7f835bfff640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 legacy=0x7f834000c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T17:56:08.435 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.435+0000 7f8360d6b640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f8340005e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T17:56:08.436 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.435+0000 7f836156c640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f8340008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T17:56:08.436 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.435+0000 7f835bfff640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 legacy=0x7f834000c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T17:56:08.836 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.836+0000 7f8360d6b640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f8340005e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T17:56:08.837 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.836+0000 7f836156c640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f8340008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T17:56:08.837 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:08.836+0000 7f835bfff640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 legacy=0x7f834000c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T17:56:09.151 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:00:06.900 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:56:09.458 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:00:07.207 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:56:09.589 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:00:07.382 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:56:09.591 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:00:07.384 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:56:09.637 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:09.637+0000 7f836156c640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f8340008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6790/0 2024-03-25T17:56:09.637 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:09.637+0000 7f8360d6b640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f8340005e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.18:6789/0 2024-03-25T17:56:09.638 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:09.637+0000 7f835bfff640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 legacy=0x7f834000c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.169:6789/0 2024-03-25T17:56:09.916 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:00:07.709 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:56:09.918 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:00:07.711 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:56:11.235 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.234+0000 7f835a7fc640 10 monclient(hunting): tick 2024-03-25T17:56:11.235 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.234+0000 7f835a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:56:11.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.234+0000 7f835a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T17:56:11.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.234+0000 7f835a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:56:11.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.234+0000 7f835a7fc640 1 -- >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f8340008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:56:11.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.234+0000 7f835a7fc640 1 -- >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 legacy=0x7f834000c5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:56:11.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.234+0000 7f835a7fc640 1 -- >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f8340005e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:56:11.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.234+0000 7f835a7fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:56:11.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.234+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f835c0a9920 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:11.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.234+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f835c0b45a0 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:11.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.234+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f835c146450 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:11.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.234+0000 7f835a7fc640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c13fec0 con 0x7f835c146450 2024-03-25T17:56:11.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.234+0000 7f835a7fc640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340010e20 con 0x7f835c0a9920 2024-03-25T17:56:11.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.234+0000 7f835a7fc640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c06b100 con 0x7f835c0b45a0 2024-03-25T17:56:11.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.234+0000 7f835a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:11.284 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.283+0000 7f8360d6b640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 0x7f83400056a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:59884/0 (socket says 172.21.15.18:59884) 2024-03-25T17:56:11.284 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.283+0000 7f8360d6b640 1 -- 172.21.15.18:0/1091891657 learned_addr learned my addr 172.21.15.18:0/1091891657 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:56:11.636 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.636+0000 7f835bfff640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f8340008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T17:56:11.637 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.636+0000 7f835bfff640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f8340008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T17:56:11.637 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.636+0000 7f835bfff640 1 --1- 172.21.15.18:0/1091891657 >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 0x7f8340008ea0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T17:56:11.695 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.694+0000 7f836156c640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f83400065d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T17:56:11.695 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.694+0000 7f836156c640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f83400065d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T17:56:11.695 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.694+0000 7f836156c640 1 --1- 172.21.15.18:0/1091891657 >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 0x7f83400065d0 :-1 s=OPENED pgs=2 cs=1 l=1).handle_message read tag failed 2024-03-25T17:56:11.696 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.694+0000 7f836156c640 1 --1- 172.21.15.18:0/1091891657 >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 0x7f83400065d0 :-1 s=OPENED pgs=2 cs=1 l=1).fault on lossy channel, failing 2024-03-25T17:56:11.696 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:11.694+0000 7f835b7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.18:6789/0 2024-03-25T17:56:14.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:14.235+0000 7f835a7fc640 10 monclient(hunting): tick 2024-03-25T17:56:14.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:14.235+0000 7f835a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:56:14.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:14.235+0000 7f835a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T17:56:14.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:14.235+0000 7f835a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:56:14.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:14.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f8340008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:14.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:14.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 legacy=0x7f83400056a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:14.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:14.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f83400065d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T17:56:14.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:14.235+0000 7f835a7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:56:14.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:14.235+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f835c146450 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:14.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:14.235+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f835c0a9920 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:14.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:14.235+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f835c0b45a0 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:14.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:14.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c0b2940 con 0x7f835c146450 2024-03-25T17:56:14.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:14.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340005e80 con 0x7f835c0a9920 2024-03-25T17:56:14.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:14.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83400060b0 con 0x7f835c0b45a0 2024-03-25T17:56:14.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:14.235+0000 7f835a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:17.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:17.235+0000 7f835a7fc640 10 monclient(hunting): tick 2024-03-25T17:56:17.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:17.235+0000 7f835a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:56:17.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:17.235+0000 7f835a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T17:56:17.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:17.235+0000 7f835a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:56:17.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:17.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.18:6790/0 conn(0x7f835c0b45a0 legacy=0x7f8340008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:17.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:17.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.169:6789/0 conn(0x7f835c0a9920 legacy=0x7f83400056a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:17.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:17.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f83400065d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:17.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:17.235+0000 7f835a7fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:56:17.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:17.235+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f834000a5e0 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:17.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:17.235+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f83400127b0 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:17.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:17.235+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f835c146450 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:17.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:17.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83400062e0 con 0x7f835c146450 2024-03-25T17:56:17.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:17.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340005770 con 0x7f83400127b0 2024-03-25T17:56:17.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:17.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83400059a0 con 0x7f834000a5e0 2024-03-25T17:56:17.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:17.235+0000 7f835a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:20.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:20.235+0000 7f835a7fc640 10 monclient(hunting): tick 2024-03-25T17:56:20.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:20.235+0000 7f835a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:56:20.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:20.235+0000 7f835a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T17:56:20.236 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:20.235+0000 7f835a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:56:20.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:20.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.18:6790/0 conn(0x7f834000a5e0 legacy=0x7f8340011090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:20.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:20.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.169:6789/0 conn(0x7f83400127b0 legacy=0x7f8340014ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:20.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:20.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.18:6789/0 conn(0x7f835c146450 legacy=0x7f83400065d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:20.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:20.235+0000 7f835a7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:56:20.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:20.235+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f834000cd20 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:20.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:20.235+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f834000d900 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:20.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:20.235+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f834000a5e0 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:20.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:20.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340005bd0 con 0x7f834000cd20 2024-03-25T17:56:20.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:20.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340006620 con 0x7f834000d900 2024-03-25T17:56:20.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:20.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340006850 con 0x7f834000a5e0 2024-03-25T17:56:20.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:20.235+0000 7f835a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:23.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:23.235+0000 7f835a7fc640 10 monclient(hunting): tick 2024-03-25T17:56:23.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:23.235+0000 7f835a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:56:23.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:23.235+0000 7f835a7fc640 1 monclient(hunting): continuing hunt 2024-03-25T17:56:23.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:23.235+0000 7f835a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:56:23.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:23.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.18:6790/0 conn(0x7f834000a5e0 legacy=0x7f8340009780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:23.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:23.235+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.169:6789/0 conn(0x7f834000d900 legacy=0x7f8340008ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:23.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:23.236+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.18:6789/0 conn(0x7f834000cd20 legacy=0x7f834000d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:23.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:23.236+0000 7f835a7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:56:23.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:23.236+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f8340009e90 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:23.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:23.236+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f834000d900 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:23.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:23.236+0000 7f835a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f834000a5e0 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:23.237 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:23.236+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340006a80 con 0x7f8340009e90 2024-03-25T17:56:23.238 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:23.236+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834000a270 con 0x7f834000d900 2024-03-25T17:56:23.238 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:23.236+0000 7f835a7fc640 1 -- 172.21.15.18:0/1091891657 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f834000d100 con 0x7f834000a5e0 2024-03-25T17:56:23.238 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:23.236+0000 7f835a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:25.933 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.932+0000 7f835b7fe640 1 -- 172.21.15.18:0/1091891657 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058695294 0 0) 0x7f8340006a80 con 0x7f8340009e90 2024-03-25T17:56:25.933 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.932+0000 7f835b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:56:25.933 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.932+0000 7f835b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:25.933 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.932+0000 7f835b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:25.933 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.932+0000 7f835b7fe640 10 monclient(hunting): my global_id is 4101 2024-03-25T17:56:25.933 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.932+0000 7f835b7fe640 1 -- 172.21.15.18:0/1091891657 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f834c003520 con 0x7f8340009e90 2024-03-25T17:56:25.947 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.946+0000 7f835b7fe640 1 -- 172.21.15.18:0/1091891657 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820249686 0 0) 0x7f834000a270 con 0x7f834000d900 2024-03-25T17:56:25.947 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.946+0000 7f835b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:56:25.947 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.946+0000 7f835b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:25.947 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.946+0000 7f835b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:25.947 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.946+0000 7f835b7fe640 10 monclient(hunting): my global_id is 4105 2024-03-25T17:56:25.947 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.946+0000 7f835b7fe640 1 -- 172.21.15.18:0/1091891657 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8340006a80 con 0x7f834000d900 2024-03-25T17:56:25.958 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.957+0000 7f835b7fe640 1 -- 172.21.15.18:0/1091891657 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277297944 0 0) 0x7f834000d100 con 0x7f834000a5e0 2024-03-25T17:56:25.958 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.957+0000 7f835b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:56:25.958 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.957+0000 7f835b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:25.958 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.957+0000 7f835b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:25.958 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.957+0000 7f835b7fe640 10 monclient(hunting): my global_id is 4106 2024-03-25T17:56:25.958 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.958+0000 7f835b7fe640 1 -- 172.21.15.18:0/1091891657 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f834000a270 con 0x7f834000a5e0 2024-03-25T17:56:25.994 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.993+0000 7f835b7fe640 1 -- 172.21.15.18:0/1091891657 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1177501289 0 0) 0x7f834c003520 con 0x7f8340009e90 2024-03-25T17:56:25.995 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.993+0000 7f835b7fe640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.18:6790/0 conn(0x7f834000a5e0 legacy=0x7f8340009780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:25.995 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.993+0000 7f835b7fe640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.169:6789/0 conn(0x7f834000d900 legacy=0x7f83400120a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:25.995 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.993+0000 7f835b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:56:25.995 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.993+0000 7f835b7fe640 1 monclient: found mon.noname-a 2024-03-25T17:56:25.995 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.993+0000 7f835b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:25.995 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.993+0000 7f835b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T17:56:25.995 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.993+0000 7f835b7fe640 1 -- 172.21.15.18:0/1091891657 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f834000d290 con 0x7f8340009e90 2024-03-25T17:56:25.995 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.993+0000 7f835b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:56:25.995 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.993+0000 7f835b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:56:25.995 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.993+0000 7f835b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:25.995 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.993+0000 7f835b7fe640 1 -- 172.21.15.18:0/1091891657 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8350004110 con 0x7f8340009e90 2024-03-25T17:56:25.995 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.993+0000 7f835b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:25.995 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.994+0000 7f835b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:56:25.996 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.994+0000 7f835b7fe640 10 monclient: dump: 2024-03-25T17:56:25.996 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-25T17:56:25.996 INFO:teuthology.orchestra.run.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:25.996 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:25.996 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:25.996 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:25.996 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:25.996 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:25.996 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:25.996 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:25.996 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T17:56:25.996 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.994+0000 7f8362fcf640 5 monclient: authenticate success, global_id 4101 2024-03-25T17:56:25.996 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:25.994+0000 7f8362fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:26.041 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.041+0000 7f835b7fe640 1 -- 172.21.15.18:0/1091891657 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83500042e0 con 0x7f8340009e90 2024-03-25T17:56:26.042 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.041+0000 7f835b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:26.042 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.041+0000 7f835b7fe640 1 -- 172.21.15.18:0/1091891657 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f83500046b0 con 0x7f8340009e90 2024-03-25T17:56:26.042 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.041+0000 7f835b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:26.042 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.041+0000 7f835b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:26.042 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.041+0000 7f835b7fe640 10 monclient: dump: 2024-03-25T17:56:26.042 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-25T17:56:26.042 INFO:teuthology.orchestra.run.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:26.042 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:26.042 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:26.042 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:26.042 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:26.042 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:26.042 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:26.043 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:26.043 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T17:56:26.043 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.041+0000 7f8362fcf640 10 monclient: get_monmap_and_config success 2024-03-25T17:56:26.043 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.041+0000 7f8362fcf640 4 set_mon_vals no callback set 2024-03-25T17:56:26.045 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.041+0000 7f8362fcf640 10 monclient: shutdown 2024-03-25T17:56:26.045 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.041+0000 7f8362fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:26.045 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.041+0000 7f8362fcf640 1 -- 172.21.15.18:0/1091891657 >> v1:172.21.15.18:6789/0 conn(0x7f8340009e90 legacy=0x7f834000fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:26.046 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.041+0000 7f8362fcf640 1 -- 172.21.15.18:0/1091891657 shutdown_connections 2024-03-25T17:56:26.046 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.041+0000 7f8362fcf640 1 -- 172.21.15.18:0/1091891657 >> 172.21.15.18:0/1091891657 conn(0x7f835c0b7f40 msgr2=0x7f835c0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:26.046 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.041+0000 7f8362fcf640 1 -- 172.21.15.18:0/1091891657 shutdown_connections 2024-03-25T17:56:26.046 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.041+0000 7f8362fcf640 1 -- 172.21.15.18:0/1091891657 wait complete. 2024-03-25T17:56:26.046 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.042+0000 7f8362fcf640 10 monclient: build_initial_monmap 2024-03-25T17:56:26.046 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.042+0000 7f8362fcf640 10 monclient: monmap: 2024-03-25T17:56:26.046 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 0 2024-03-25T17:56:26.046 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:56:26.046 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 0.000000 2024-03-25T17:56:26.046 INFO:teuthology.orchestra.run.smithi018.stderr:created 0.000000 2024-03-25T17:56:26.046 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:26.046 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-25T17:56:26.046 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:26.046 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:56:26.047 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:56:26.047 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T17:56:26.047 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.042+0000 7f8362fcf640 1 librados: starting msgr at 2024-03-25T17:56:26.047 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.042+0000 7f8362fcf640 1 librados: starting objecter 2024-03-25T17:56:26.047 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.042+0000 7f8362fcf640 1 Processor -- start 2024-03-25T17:56:26.047 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.043+0000 7f8362fcf640 1 -- start start 2024-03-25T17:56:26.047 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.043+0000 7f8362fcf640 1 librados: setting wanted keys 2024-03-25T17:56:26.047 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.043+0000 7f8362fcf640 1 librados: calling monclient init 2024-03-25T17:56:26.047 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.043+0000 7f8362fcf640 10 monclient: init 2024-03-25T17:56:26.047 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.043+0000 7f8362fcf640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:26.047 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.043+0000 7f8362fcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:56:26.047 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.043+0000 7f8362fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f8340009e90 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:26.047 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.043+0000 7f8362fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f834000a5e0 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:26.047 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.043+0000 7f8362fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f834000d900 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:26.048 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.043+0000 7f8362fcf640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c0ad580 con 0x7f834000a5e0 2024-03-25T17:56:26.048 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.043+0000 7f8362fcf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c1d7440 con 0x7f834000d900 2024-03-25T17:56:26.048 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.043+0000 7f8362fcf640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835c1d8070 con 0x7f8340009e90 2024-03-25T17:56:26.048 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.043+0000 7f8362fcf640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:26.048 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.043+0000 7f8362fcf640 10 monclient(hunting): authenticate will time out at 6914.622477s 2024-03-25T17:56:26.048 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.044+0000 7f8360d6b640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f8340009e90 0x7f835c1d7960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:54162/0 (socket says 172.21.15.18:54162) 2024-03-25T17:56:26.048 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.044+0000 7f8360d6b640 1 -- 172.21.15.18:0/4199348184 learned_addr learned my addr 172.21.15.18:0/4199348184 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:56:26.048 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.044+0000 7f83597fa640 10 client.?.objecter ms_handle_connect 0x7f834000d900 2024-03-25T17:56:26.048 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.044+0000 7f83597fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:26.048 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.044+0000 7f83597fa640 10 client.?.objecter ms_handle_connect 0x7f8340009e90 2024-03-25T17:56:26.048 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.044+0000 7f83597fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:26.049 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.049+0000 7f83597fa640 10 client.?.objecter ms_handle_connect 0x7f834000a5e0 2024-03-25T17:56:26.049 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.049+0000 7f83597fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:26.096 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.095+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378259176 0 0) 0x7f835c0ad580 con 0x7f834000a5e0 2024-03-25T17:56:26.096 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.095+0000 7f83597fa640 10 monclient(hunting): _negotiate 2024-03-25T17:56:26.096 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.095+0000 7f83597fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:26.096 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.095+0000 7f83597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:26.096 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.095+0000 7f83597fa640 10 monclient(hunting): my global_id is 4110 2024-03-25T17:56:26.096 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.095+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f834000d420 con 0x7f834000a5e0 2024-03-25T17:56:26.110 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.109+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931035314 0 0) 0x7f835c1d8070 con 0x7f8340009e90 2024-03-25T17:56:26.110 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.109+0000 7f83597fa640 10 monclient(hunting): _negotiate 2024-03-25T17:56:26.110 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.109+0000 7f83597fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:26.110 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.109+0000 7f83597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:26.110 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.109+0000 7f83597fa640 10 monclient(hunting): my global_id is 4112 2024-03-25T17:56:26.111 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.109+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f835c0ad580 con 0x7f8340009e90 2024-03-25T17:56:26.116 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.116+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337795457 0 0) 0x7f835c1d7440 con 0x7f834000d900 2024-03-25T17:56:26.117 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.116+0000 7f83597fa640 10 monclient(hunting): _negotiate 2024-03-25T17:56:26.117 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.116+0000 7f83597fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:26.117 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.116+0000 7f83597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:26.117 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.116+0000 7f83597fa640 10 monclient(hunting): my global_id is 4111 2024-03-25T17:56:26.117 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.116+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f835c1d8070 con 0x7f834000d900 2024-03-25T17:56:26.123 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.122+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2348296132 0 0) 0x7f834000d420 con 0x7f834000a5e0 2024-03-25T17:56:26.123 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.122+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f835c1d7440 con 0x7f834000a5e0 2024-03-25T17:56:26.123 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.122+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8348004250 con 0x7f834000a5e0 2024-03-25T17:56:26.123 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.122+0000 7f83597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:26.123 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.122+0000 7f83597fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:56:26.123 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.122+0000 7f83597fa640 10 monclient(hunting): dump: 2024-03-25T17:56:26.123 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-25T17:56:26.123 INFO:teuthology.orchestra.run.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:26.123 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:26.123 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:26.123 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:26.123 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:26.124 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:26.124 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:26.124 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:26.124 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T17:56:26.124 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.122+0000 7f83597fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:56:26.228 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.226+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1279505688 0 0) 0x7f835c1d7440 con 0x7f834000a5e0 2024-03-25T17:56:26.228 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.226+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 >> v1:172.21.15.18:6790/0 conn(0x7f8340009e90 legacy=0x7f835c1d7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:26.228 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.226+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 >> v1:172.21.15.169:6789/0 conn(0x7f834000d900 legacy=0x7f835c1e2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:26.228 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.226+0000 7f83597fa640 10 monclient: _finish_hunting 0 2024-03-25T17:56:26.228 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.226+0000 7f83597fa640 1 monclient: found mon.a 2024-03-25T17:56:26.228 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.226+0000 7f83597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:26.228 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.226+0000 7f83597fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:26.228 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.226+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835c1d8200 con 0x7f834000a5e0 2024-03-25T17:56:26.228 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.226+0000 7f83597fa640 10 monclient: _finish_auth 0 2024-03-25T17:56:26.228 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.226+0000 7f83597fa640 10 monclient: _check_auth_tickets 2024-03-25T17:56:26.228 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.226+0000 7f83597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:26.229 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.227+0000 7f8362fcf640 5 monclient: authenticate success, global_id 4110 2024-03-25T17:56:26.229 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.227+0000 7f8362fcf640 10 monclient: _renew_subs 2024-03-25T17:56:26.229 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.227+0000 7f8362fcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:26.229 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.227+0000 7f8362fcf640 1 -- 172.21.15.18:0/4199348184 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f835c138930 con 0x7f834000a5e0 2024-03-25T17:56:26.229 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.227+0000 7f8362fcf640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:56:26.229 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.227+0000 7f8362fcf640 10 monclient: _renew_subs 2024-03-25T17:56:26.229 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.227+0000 7f8362fcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:26.229 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.227+0000 7f8362fcf640 1 -- 172.21.15.18:0/4199348184 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f835c140910 con 0x7f834000a5e0 2024-03-25T17:56:26.229 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.227+0000 7f8362fcf640 1 librados: init done 2024-03-25T17:56:26.230 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.228+0000 7f8362fcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:26.230 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.228+0000 7f8362fcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:26.230 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.228+0000 7f8362fcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:26.230 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.228+0000 7f8362fcf640 1 -- 172.21.15.18:0/4199348184 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f835c140910 con 0x7f834000a5e0 2024-03-25T17:56:26.497 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.496+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83480046d0 con 0x7f834000a5e0 2024-03-25T17:56:26.497 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.496+0000 7f83597fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:26.497 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.496+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8348004a60 con 0x7f834000a5e0 2024-03-25T17:56:26.498 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.496+0000 7f83597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:26.498 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.496+0000 7f83597fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:26.498 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.496+0000 7f83597fa640 10 monclient: dump: 2024-03-25T17:56:26.498 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-25T17:56:26.498 INFO:teuthology.orchestra.run.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:26.498 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:26.498 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:26.498 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:26.498 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:26.498 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:26.498 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:26.498 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:26.498 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T17:56:26.499 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.496+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8348005010 con 0x7f834000a5e0 2024-03-25T17:56:26.499 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.496+0000 7f835affd640 4 set_mon_vals no callback set 2024-03-25T17:56:26.499 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.496+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3920291517 0 0) 0x7f8348006510 con 0x7f834000a5e0 2024-03-25T17:56:26.499 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.496+0000 7f83597fa640 10 client.4110.objecter ms_dispatch 0x7f835c000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T17:56:26.499 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.496+0000 7f83597fa640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T17:56:26.499 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.496+0000 7f83597fa640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-25T17:56:26.499 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.496+0000 7f83597fa640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-25T17:56:26.499 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.496+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8348006950 con 0x7f834000a5e0 2024-03-25T17:56:26.499 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.496+0000 7f83597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:26.499 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.496+0000 7f83597fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:56:26.614 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.613+0000 7f8362fcf640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T17:56:26.614 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.613+0000 7f8362fcf640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T17:56:26.614 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.613+0000 7f8362fcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:26.614 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.613+0000 7f8362fcf640 1 -- 172.21.15.18:0/4199348184 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f835c1d7650 con 0x7f834000a5e0 2024-03-25T17:56:26.769 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.769+0000 7f83597fa640 1 -- 172.21.15.18:0/4199348184 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f83480055d0 con 0x7f834000a5e0 2024-03-25T17:56:26.770 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.769+0000 7f83597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T17:56:26.770 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.769+0000 7f83597fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T17:56:26.770 INFO:teuthology.orchestra.run.smithi018.stderr:adjusted tunables profile to default 2024-03-25T17:56:26.771 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.770+0000 7f8362fcf640 10 librados: watch_flush enter 2024-03-25T17:56:26.771 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.770+0000 7f8362fcf640 10 librados: watch_flush exit 2024-03-25T17:56:26.771 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.770+0000 7f8362fcf640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-25T17:56:26.771 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.770+0000 7f8362fcf640 10 client.4110.objecter successfully canceled tick 2024-03-25T17:56:26.771 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.770+0000 7f8362fcf640 10 monclient: shutdown 2024-03-25T17:56:26.771 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.770+0000 7f8362fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:26.772 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.770+0000 7f8362fcf640 1 -- 172.21.15.18:0/4199348184 >> v1:172.21.15.18:6789/0 conn(0x7f834000a5e0 legacy=0x7f835c138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:26.772 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.770+0000 7f8362fcf640 1 -- 172.21.15.18:0/4199348184 shutdown_connections 2024-03-25T17:56:26.772 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.770+0000 7f8362fcf640 1 -- 172.21.15.18:0/4199348184 >> 172.21.15.18:0/4199348184 conn(0x7f835c0b7f40 msgr2=0x7f835c0b5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:26.772 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.770+0000 7f8362fcf640 1 -- 172.21.15.18:0/4199348184 shutdown_connections 2024-03-25T17:56:26.772 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.770+0000 7f8362fcf640 1 -- 172.21.15.18:0/4199348184 wait complete. 2024-03-25T17:56:26.772 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:26.770+0000 7f8362fcf640 1 librados: shutdown 2024-03-25T17:56:26.786 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T17:56:26.786 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T17:56:26.786 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:56:26.786 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T17:56:26.789 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:26.788+0000 7fc44ee54040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T17:56:26.813 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:26.813+0000 7fb497bf1040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T17:56:26.815 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:56:26.815 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T17:56:26.851 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:26.851+0000 7fc44ee54040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T17:56:26.868 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:26.868+0000 7fb497bf1040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T17:56:26.879 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:56:26.879 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T17:56:26.948 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-25T17:56:26.949 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T17:56:27.016 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T17:56:27.016 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T17:56:27.044 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T17:56:27.044 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T17:56:27.114 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T17:56:27.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T17:56:27.180 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-03-25T17:56:27.180 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T17:56:27.247 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 4f033a33-dc7d-4490-8b69-56479f880977 0 2024-03-25T17:56:27.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.387+0000 7f9c6399d640 10 monclient: get_monmap_and_config 2024-03-25T17:56:27.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.387+0000 7f9c6399d640 10 monclient: build_initial_monmap 2024-03-25T17:56:27.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.387+0000 7f9c6399d640 10 monclient: monmap: 2024-03-25T17:56:27.388 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T17:56:27.388 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:27.388 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:56:27.388459+0000 2024-03-25T17:56:27.388 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:56:27.388459+0000 2024-03-25T17:56:27.388 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:27.388 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:27.388 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:27.388 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:56:27.388 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:56:27.388 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:27.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.388+0000 7f9c6399d640 1 Processor -- start 2024-03-25T17:56:27.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.388+0000 7f9c6399d640 1 -- start start 2024-03-25T17:56:27.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.388+0000 7f9c6399d640 10 monclient: init 2024-03-25T17:56:27.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.388+0000 7f9c6399d640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:27.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.388+0000 7f9c6399d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:56:27.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.388+0000 7f9c6399d640 10 monclient(hunting): picked mon.noname-b con 0x7f9c5c0b4700 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:27.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.388+0000 7f9c6399d640 10 monclient(hunting): picked mon.noname-a con 0x7f9c5c142a20 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:27.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.388+0000 7f9c6399d640 10 monclient(hunting): picked mon.noname-c con 0x7f9c5c0a96b0 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:27.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.388+0000 7f9c6399d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c0a9d60 con 0x7f9c5c142a20 2024-03-25T17:56:27.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.388+0000 7f9c6399d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c13ff10 con 0x7f9c5c0a96b0 2024-03-25T17:56:27.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.388+0000 7f9c6399d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c0aa1f0 con 0x7f9c5c0b4700 2024-03-25T17:56:27.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.388+0000 7f9c6399d640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:27.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.388+0000 7f9c6399d640 10 monclient(hunting): authenticate will time out at 6902.674256s 2024-03-25T17:56:27.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.389+0000 7f9c61f3a640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f9c5c0a96b0 0x7f9c5c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:50110/0 (socket says 172.21.15.169:50110) 2024-03-25T17:56:27.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.389+0000 7f9c61f3a640 1 -- 172.21.15.169:0/2988128902 learned_addr learned my addr 172.21.15.169:0/2988128902 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T17:56:27.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.391+0000 7f9c53fff640 1 -- 172.21.15.169:0/2988128902 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3282440038 0 0) 0x7f9c5c13ff10 con 0x7f9c5c0a96b0 2024-03-25T17:56:27.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.391+0000 7f9c53fff640 10 monclient(hunting): _negotiate 2024-03-25T17:56:27.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.391+0000 7f9c53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:27.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.391+0000 7f9c53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:27.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.391+0000 7f9c53fff640 10 monclient(hunting): my global_id is 4117 2024-03-25T17:56:27.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.391+0000 7f9c53fff640 1 -- 172.21.15.169:0/2988128902 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c48003520 con 0x7f9c5c0a96b0 2024-03-25T17:56:27.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.392+0000 7f9c53fff640 1 -- 172.21.15.169:0/2988128902 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535820539 0 0) 0x7f9c5c0a9d60 con 0x7f9c5c142a20 2024-03-25T17:56:27.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.392+0000 7f9c53fff640 10 monclient(hunting): _negotiate 2024-03-25T17:56:27.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.392+0000 7f9c53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:27.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.392+0000 7f9c53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:27.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.392+0000 7f9c53fff640 10 monclient(hunting): my global_id is 4116 2024-03-25T17:56:27.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.392+0000 7f9c53fff640 1 -- 172.21.15.169:0/2988128902 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5c13ff10 con 0x7f9c5c142a20 2024-03-25T17:56:27.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.393+0000 7f9c53fff640 1 -- 172.21.15.169:0/2988128902 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683842603 0 0) 0x7f9c5c0aa1f0 con 0x7f9c5c0b4700 2024-03-25T17:56:27.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.393+0000 7f9c53fff640 10 monclient(hunting): _negotiate 2024-03-25T17:56:27.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.393+0000 7f9c53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:27.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.393+0000 7f9c53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:27.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.393+0000 7f9c53fff640 10 monclient(hunting): my global_id is 4118 2024-03-25T17:56:27.393 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.393+0000 7f9c53fff640 1 -- 172.21.15.169:0/2988128902 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5c0a9d60 con 0x7f9c5c0b4700 2024-03-25T17:56:27.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.394+0000 7f9c53fff640 1 -- 172.21.15.169:0/2988128902 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 501600194 0 0) 0x7f9c48003520 con 0x7f9c5c0a96b0 2024-03-25T17:56:27.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.394+0000 7f9c53fff640 1 -- 172.21.15.169:0/2988128902 >> v1:172.21.15.18:6790/0 conn(0x7f9c5c0b4700 legacy=0x7f9c5c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:27.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.394+0000 7f9c53fff640 1 -- 172.21.15.169:0/2988128902 >> v1:172.21.15.18:6789/0 conn(0x7f9c5c142a20 legacy=0x7f9c5c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:27.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.394+0000 7f9c53fff640 10 monclient: _finish_hunting 0 2024-03-25T17:56:27.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.394+0000 7f9c53fff640 1 monclient: found mon.noname-c 2024-03-25T17:56:27.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.394+0000 7f9c53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:27.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.394+0000 7f9c53fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T17:56:27.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.394+0000 7f9c53fff640 1 -- 172.21.15.169:0/2988128902 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c0b4ae0 con 0x7f9c5c0a96b0 2024-03-25T17:56:27.395 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.394+0000 7f9c53fff640 10 monclient: _finish_auth 0 2024-03-25T17:56:27.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.394+0000 7f9c53fff640 10 monclient: _check_auth_tickets 2024-03-25T17:56:27.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.394+0000 7f9c53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:27.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.394+0000 7f9c53fff640 1 -- 172.21.15.169:0/2988128902 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9c54001a10 con 0x7f9c5c0a96b0 2024-03-25T17:56:27.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.394+0000 7f9c6399d640 5 monclient: authenticate success, global_id 4117 2024-03-25T17:56:27.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.394+0000 7f9c6399d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:27.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.394+0000 7f9c53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:27.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.395+0000 7f9c53fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:56:27.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.395+0000 7f9c53fff640 10 monclient: dump: 2024-03-25T17:56:27.397 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:27.397 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:27.397 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.397 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.397 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:27.397 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:27.397 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:27.397 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:27.397 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:27.397 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:27.397 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.395+0000 7f9c6399d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:27.400 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.397+0000 7f9c53fff640 1 -- 172.21.15.169:0/2988128902 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c54001b70 con 0x7f9c5c0a96b0 2024-03-25T17:56:27.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.397+0000 7f9c53fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:27.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.397+0000 7f9c53fff640 1 -- 172.21.15.169:0/2988128902 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9c54001f80 con 0x7f9c5c0a96b0 2024-03-25T17:56:27.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.397+0000 7f9c53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:27.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.397+0000 7f9c53fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:56:27.401 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.397+0000 7f9c53fff640 10 monclient: dump: 2024-03-25T17:56:27.401 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:27.401 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:27.401 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.401 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.401 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:27.401 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:27.401 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:27.401 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:27.402 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:27.402 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:27.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.397+0000 7f9c6399d640 10 monclient: get_monmap_and_config success 2024-03-25T17:56:27.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.397+0000 7f9c6399d640 4 set_mon_vals no callback set 2024-03-25T17:56:27.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.398+0000 7f9c6399d640 10 monclient: shutdown 2024-03-25T17:56:27.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.398+0000 7f9c6399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:27.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.398+0000 7f9c6399d640 1 -- 172.21.15.169:0/2988128902 >> v1:172.21.15.169:6789/0 conn(0x7f9c5c0a96b0 legacy=0x7f9c5c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:27.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.398+0000 7f9c6399d640 1 -- 172.21.15.169:0/2988128902 shutdown_connections 2024-03-25T17:56:27.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.398+0000 7f9c6399d640 1 -- 172.21.15.169:0/2988128902 >> 172.21.15.169:0/2988128902 conn(0x7f9c5c0668e0 msgr2=0x7f9c5c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:27.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.398+0000 7f9c6399d640 1 -- 172.21.15.169:0/2988128902 shutdown_connections 2024-03-25T17:56:27.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.398+0000 7f9c6399d640 1 -- 172.21.15.169:0/2988128902 wait complete. 2024-03-25T17:56:27.402 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.398+0000 7f9c6399d640 10 monclient: build_initial_monmap 2024-03-25T17:56:27.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.398+0000 7f9c6399d640 10 monclient: monmap: 2024-03-25T17:56:27.403 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T17:56:27.403 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:56:27.403 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-25T17:56:27.403 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-25T17:56:27.403 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:27.403 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-25T17:56:27.403 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:27.403 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:56:27.403 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:56:27.403 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:27.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.399+0000 7f9c6399d640 1 librados: starting msgr at 2024-03-25T17:56:27.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.399+0000 7f9c6399d640 1 librados: starting objecter 2024-03-25T17:56:27.403 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.399+0000 7f9c6399d640 1 Processor -- start 2024-03-25T17:56:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.399+0000 7f9c6399d640 1 -- start start 2024-03-25T17:56:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.399+0000 7f9c6399d640 1 librados: setting wanted keys 2024-03-25T17:56:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.399+0000 7f9c6399d640 1 librados: calling monclient init 2024-03-25T17:56:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.399+0000 7f9c6399d640 10 monclient: init 2024-03-25T17:56:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.399+0000 7f9c6399d640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.399+0000 7f9c6399d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:56:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.399+0000 7f9c6399d640 10 monclient(hunting): picked mon.noname-b con 0x7f9c5c0a96b0 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.399+0000 7f9c6399d640 10 monclient(hunting): picked mon.noname-c con 0x7f9c5c0b4700 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.399+0000 7f9c6399d640 10 monclient(hunting): picked mon.noname-a con 0x7f9c5c142a20 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.399+0000 7f9c6399d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c13b110 con 0x7f9c5c142a20 2024-03-25T17:56:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.399+0000 7f9c6399d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c1dbe60 con 0x7f9c5c0a96b0 2024-03-25T17:56:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.399+0000 7f9c6399d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c0abc90 con 0x7f9c5c0b4700 2024-03-25T17:56:27.404 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.399+0000 7f9c6399d640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.399+0000 7f9c6399d640 10 monclient(hunting): authenticate will time out at 6902.685322s 2024-03-25T17:56:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.400+0000 7f9c61f3a640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f9c5c142a20 0x7f9c5c1db750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.169:44018/0 (socket says 172.21.15.169:44018) 2024-03-25T17:56:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.400+0000 7f9c61739640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f9c5c0a96b0 0x7f9c5c1db040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:50126/0 (socket says 172.21.15.169:50126) 2024-03-25T17:56:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.400+0000 7f9c61f3a640 1 -- 172.21.15.169:0/1430902100 learned_addr learned my addr 172.21.15.169:0/1430902100 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T17:56:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.400+0000 7f9c51ffb640 10 client.?.objecter ms_handle_connect 0x7f9c5c0a96b0 2024-03-25T17:56:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.400+0000 7f9c51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.400+0000 7f9c51ffb640 10 client.?.objecter ms_handle_connect 0x7f9c5c142a20 2024-03-25T17:56:27.405 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.400+0000 7f9c51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:27.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.401+0000 7f9c51ffb640 10 client.?.objecter ms_handle_connect 0x7f9c5c0b4700 2024-03-25T17:56:27.406 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.401+0000 7f9c51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.402+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765188578 0 0) 0x7f9c5c1dbe60 con 0x7f9c5c0a96b0 2024-03-25T17:56:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.402+0000 7f9c51ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:56:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.402+0000 7f9c51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.402+0000 7f9c51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.402+0000 7f9c51ffb640 10 monclient(hunting): my global_id is 4120 2024-03-25T17:56:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.402+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c40004eb0 con 0x7f9c5c0a96b0 2024-03-25T17:56:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.402+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327018746 0 0) 0x7f9c5c13b110 con 0x7f9c5c142a20 2024-03-25T17:56:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.402+0000 7f9c51ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:56:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.402+0000 7f9c51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.402+0000 7f9c51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.402+0000 7f9c51ffb640 10 monclient(hunting): my global_id is 4119 2024-03-25T17:56:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.402+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5c1dbe60 con 0x7f9c5c142a20 2024-03-25T17:56:27.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.403+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318004176 0 0) 0x7f9c5c0abc90 con 0x7f9c5c0b4700 2024-03-25T17:56:27.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.403+0000 7f9c51ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:56:27.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.403+0000 7f9c51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:27.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.403+0000 7f9c51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:27.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.403+0000 7f9c51ffb640 10 monclient(hunting): my global_id is 4121 2024-03-25T17:56:27.408 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.403+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5c13b110 con 0x7f9c5c0b4700 2024-03-25T17:56:27.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.407+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1600300115 0 0) 0x7f9c40004eb0 con 0x7f9c5c0a96b0 2024-03-25T17:56:27.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.407+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c5c0abc90 con 0x7f9c5c0a96b0 2024-03-25T17:56:27.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.407+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9c4c001970 con 0x7f9c5c0a96b0 2024-03-25T17:56:27.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.407+0000 7f9c51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:27.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.407+0000 7f9c51ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:56:27.409 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.407+0000 7f9c51ffb640 10 monclient(hunting): dump: 2024-03-25T17:56:27.409 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:27.409 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:27.409 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.409 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.409 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:27.410 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:27.410 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:27.410 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:27.410 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:27.410 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:27.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.407+0000 7f9c51ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:56:27.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.408+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2906430676 0 0) 0x7f9c5c1dbe60 con 0x7f9c5c142a20 2024-03-25T17:56:27.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.408+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c40004eb0 con 0x7f9c5c142a20 2024-03-25T17:56:27.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.408+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9c54001e80 con 0x7f9c5c142a20 2024-03-25T17:56:27.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.408+0000 7f9c51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:27.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.408+0000 7f9c51ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:27.410 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.408+0000 7f9c51ffb640 10 monclient(hunting): dump: 2024-03-25T17:56:27.410 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:27.411 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:27.411 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.411 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.411 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:27.411 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:27.411 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:27.411 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:27.411 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:27.411 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:27.412 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.411+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1459042299 0 0) 0x7f9c5c13b110 con 0x7f9c5c0b4700 2024-03-25T17:56:27.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.411+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c5c1dbe60 con 0x7f9c5c0b4700 2024-03-25T17:56:27.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.411+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9c58001930 con 0x7f9c5c0b4700 2024-03-25T17:56:27.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.411+0000 7f9c51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:27.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.411+0000 7f9c51ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:56:27.413 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.411+0000 7f9c51ffb640 10 monclient(hunting): dump: 2024-03-25T17:56:27.413 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:27.413 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:27.413 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.413 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.413 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:27.413 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:27.413 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:27.414 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:27.414 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:27.414 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:27.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4289501162 0 0) 0x7f9c40004eb0 con 0x7f9c5c142a20 2024-03-25T17:56:27.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 >> v1:172.21.15.18:6790/0 conn(0x7f9c5c0b4700 legacy=0x7f9c5c1397e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:27.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 >> v1:172.21.15.169:6789/0 conn(0x7f9c5c0a96b0 legacy=0x7f9c5c1db040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:27.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c51ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:56:27.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c51ffb640 1 monclient: found mon.a 2024-03-25T17:56:27.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:27.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c51ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:27.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c0abf70 con 0x7f9c5c142a20 2024-03-25T17:56:27.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c51ffb640 10 monclient: _finish_auth 0 2024-03-25T17:56:27.414 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c51ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:56:27.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:27.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c6399d640 5 monclient: authenticate success, global_id 4119 2024-03-25T17:56:27.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c6399d640 10 monclient: _renew_subs 2024-03-25T17:56:27.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c6399d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:27.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c6399d640 1 -- 172.21.15.169:0/1430902100 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5c0ac250 con 0x7f9c5c142a20 2024-03-25T17:56:27.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c6399d640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:56:27.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c6399d640 10 monclient: _renew_subs 2024-03-25T17:56:27.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c6399d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:27.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c6399d640 1 -- 172.21.15.169:0/1430902100 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5c14e6c0 con 0x7f9c5c142a20 2024-03-25T17:56:27.415 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.412+0000 7f9c6399d640 1 librados: init done 2024-03-25T17:56:27.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.413+0000 7f9c6399d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:27.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.413+0000 7f9c6399d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:27.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.413+0000 7f9c6399d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:27.416 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.413+0000 7f9c6399d640 1 -- 172.21.15.169:0/1430902100 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c5c13b2c0 con 0x7f9c5c142a20 2024-03-25T17:56:27.440 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:27.439+0000 7fc44ee54040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T17:56:27.453 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:27.453+0000 7fb497bf1040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T17:56:27.497 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:27.496+0000 7fc44ee54040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T17:56:27.508 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:27.508+0000 7fb497bf1040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T17:56:27.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.522+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c54001570 con 0x7f9c5c142a20 2024-03-25T17:56:27.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.522+0000 7f9c51ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:27.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.522+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9c54004e10 con 0x7f9c5c142a20 2024-03-25T17:56:27.522 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.522+0000 7f9c51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:27.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.522+0000 7f9c51ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:27.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.522+0000 7f9c51ffb640 10 monclient: dump: 2024-03-25T17:56:27.523 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:27.523 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:27.523 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.523 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.523 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:27.523 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:27.523 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:27.523 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:27.523 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:27.523 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:27.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.522+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9c540053c0 con 0x7f9c5c142a20 2024-03-25T17:56:27.523 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.522+0000 7f9c537fe640 4 set_mon_vals no callback set 2024-03-25T17:56:27.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.523+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2282660139 0 0) 0x7f9c540068a0 con 0x7f9c5c142a20 2024-03-25T17:56:27.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.523+0000 7f9c51ffb640 10 client.4119.objecter ms_dispatch 0x7f9c5c000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T17:56:27.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.523+0000 7f9c51ffb640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T17:56:27.524 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.523+0000 7f9c51ffb640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T17:56:27.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.523+0000 7f9c51ffb640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T17:56:27.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.524+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9c54006b90 con 0x7f9c5c142a20 2024-03-25T17:56:27.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.524+0000 7f9c51ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:27.525 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.524+0000 7f9c51ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:56:27.544 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:27.544+0000 7fc44ee54040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T17:56:27.554 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:27.554+0000 7fb497bf1040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T17:56:27.595 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.595+0000 7f9c6399d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4f033a33-dc7d-4490-8b69-56479f880977", "id": 0}] 2024-03-25T17:56:27.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.595+0000 7f9c6399d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4f033a33-dc7d-4490-8b69-56479f880977", "id": 0}] 2024-03-25T17:56:27.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.595+0000 7f9c6399d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:27.596 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.595+0000 7f9c6399d640 1 -- 172.21.15.169:0/1430902100 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4f033a33-dc7d-4490-8b69-56479f880977", "id": 0} v 0) v1 -- 0x7f9c5c04f730 con 0x7f9c5c142a20 2024-03-25T17:56:27.600 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:27.599+0000 7fc44ee54040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T17:56:27.608 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:27.608+0000 7fb497bf1040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T17:56:27.644 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:27.644+0000 7fc44ee54040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T17:56:27.650 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:27.650+0000 7fb497bf1040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T17:56:27.706 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.706+0000 7f9c51ffb640 1 -- 172.21.15.169:0/1430902100 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4f033a33-dc7d-4490-8b69-56479f880977", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3560235103 0 732707515) 0x7f9c54005920 con 0x7f9c5c142a20 2024-03-25T17:56:27.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.706+0000 7f9c51ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4f033a33-dc7d-4490-8b69-56479f880977", "id": 0}] 2024-03-25T17:56:27.707 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.706+0000 7f9c51ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:56:27.707 INFO:teuthology.orchestra.run.smithi169.stdout:0 2024-03-25T17:56:27.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.707+0000 7f9c6399d640 10 librados: watch_flush enter 2024-03-25T17:56:27.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.707+0000 7f9c6399d640 10 librados: watch_flush exit 2024-03-25T17:56:27.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.707+0000 7f9c6399d640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T17:56:27.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.707+0000 7f9c6399d640 10 client.4119.objecter successfully canceled tick 2024-03-25T17:56:27.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.708+0000 7f9c6399d640 10 monclient: shutdown 2024-03-25T17:56:27.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.708+0000 7f9c6399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:27.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.708+0000 7f9c6399d640 1 -- 172.21.15.169:0/1430902100 >> v1:172.21.15.18:6789/0 conn(0x7f9c5c142a20 legacy=0x7f9c5c1db750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:27.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.708+0000 7f9c6399d640 1 -- 172.21.15.169:0/1430902100 shutdown_connections 2024-03-25T17:56:27.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.708+0000 7f9c6399d640 1 -- 172.21.15.169:0/1430902100 >> 172.21.15.169:0/1430902100 conn(0x7f9c5c0668e0 msgr2=0x7f9c5c13e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:27.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.708+0000 7f9c6399d640 1 -- 172.21.15.169:0/1430902100 shutdown_connections 2024-03-25T17:56:27.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.708+0000 7f9c6399d640 1 -- 172.21.15.169:0/1430902100 wait complete. 2024-03-25T17:56:27.709 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.708+0000 7f9c6399d640 1 librados: shutdown 2024-03-25T17:56:27.721 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 2f60b59e-0fb3-4ad1-a470-9eec2498bb28 1 2024-03-25T17:56:27.771 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:27.770+0000 7fc44ee54040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T17:56:27.773 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:27.773+0000 7fb497bf1040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T17:56:27.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.821+0000 7f9f025b5640 10 monclient: get_monmap_and_config 2024-03-25T17:56:27.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.821+0000 7f9f025b5640 10 monclient: build_initial_monmap 2024-03-25T17:56:27.821 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.821+0000 7f9f025b5640 10 monclient: monmap: 2024-03-25T17:56:27.822 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T17:56:27.822 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:27.822 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:56:27.822268+0000 2024-03-25T17:56:27.822 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:56:27.822268+0000 2024-03-25T17:56:27.822 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:27.822 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:27.822 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:27.822 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:56:27.822 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:56:27.822 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:27.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.822+0000 7f9f025b5640 1 Processor -- start 2024-03-25T17:56:27.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.822+0000 7f9f025b5640 1 -- start start 2024-03-25T17:56:27.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.822+0000 7f9f025b5640 10 monclient: init 2024-03-25T17:56:27.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.822+0000 7f9f025b5640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:27.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.822+0000 7f9f025b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:56:27.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.822+0000 7f9f025b5640 10 monclient(hunting): picked mon.noname-c con 0x7f9efc070210 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:27.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.822+0000 7f9f025b5640 10 monclient(hunting): picked mon.noname-b con 0x7f9efc0b06d0 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:27.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.822+0000 7f9f025b5640 10 monclient(hunting): picked mon.noname-a con 0x7f9efc0705f0 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:27.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.822+0000 7f9f025b5640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc0b0b80 con 0x7f9efc0705f0 2024-03-25T17:56:27.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.822+0000 7f9f025b5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc0b0430 con 0x7f9efc070210 2024-03-25T17:56:27.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.822+0000 7f9f025b5640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc071670 con 0x7f9efc0b06d0 2024-03-25T17:56:27.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.822+0000 7f9f025b5640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:27.824 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.822+0000 7f9f025b5640 10 monclient(hunting): authenticate will time out at 6903.108109s 2024-03-25T17:56:27.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.823+0000 7f9efbfff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f9efc070210 0x7f9efc140930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:32978/0 (socket says 172.21.15.169:32978) 2024-03-25T17:56:27.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.823+0000 7f9efbfff640 1 -- 172.21.15.169:0/3728825764 learned_addr learned my addr 172.21.15.169:0/3728825764 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T17:56:27.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.825+0000 7f9efaffd640 1 -- 172.21.15.169:0/3728825764 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035499929 0 0) 0x7f9ef0001110 con 0x7f9efc070210 2024-03-25T17:56:27.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.825+0000 7f9efaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:56:27.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.825+0000 7f9efaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:27.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.825+0000 7f9efaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:27.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.825+0000 7f9efaffd640 10 monclient(hunting): my global_id is 4123 2024-03-25T17:56:27.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.825+0000 7f9efaffd640 1 -- 172.21.15.169:0/3728825764 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee4003520 con 0x7f9efc070210 2024-03-25T17:56:27.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.825+0000 7f9efaffd640 1 -- 172.21.15.169:0/3728825764 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220534539 0 0) 0x7f9efc071670 con 0x7f9efc0b06d0 2024-03-25T17:56:27.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.825+0000 7f9efaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:56:27.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.825+0000 7f9efaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:27.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.825+0000 7f9efaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:27.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.825+0000 7f9efaffd640 10 monclient(hunting): my global_id is 4124 2024-03-25T17:56:27.826 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.825+0000 7f9efaffd640 1 -- 172.21.15.169:0/3728825764 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ef0001110 con 0x7f9efc0b06d0 2024-03-25T17:56:27.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.826+0000 7f9efaffd640 1 -- 172.21.15.169:0/3728825764 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359607053 0 0) 0x7f9efc0b0b80 con 0x7f9efc0705f0 2024-03-25T17:56:27.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.826+0000 7f9efaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:56:27.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.826+0000 7f9efaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:27.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.826+0000 7f9efaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:27.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.826+0000 7f9efaffd640 10 monclient(hunting): my global_id is 4122 2024-03-25T17:56:27.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.826+0000 7f9efaffd640 1 -- 172.21.15.169:0/3728825764 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9efc071670 con 0x7f9efc0705f0 2024-03-25T17:56:27.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9efaffd640 1 -- 172.21.15.169:0/3728825764 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1177552257 0 0) 0x7f9ee4003520 con 0x7f9efc070210 2024-03-25T17:56:27.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9efaffd640 1 -- 172.21.15.169:0/3728825764 >> v1:172.21.15.18:6790/0 conn(0x7f9efc0b06d0 legacy=0x7f9efc0afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:27.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9efaffd640 1 -- 172.21.15.169:0/3728825764 >> v1:172.21.15.18:6789/0 conn(0x7f9efc0705f0 legacy=0x7f9efc14b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:27.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9efaffd640 10 monclient: _finish_hunting 0 2024-03-25T17:56:27.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9efaffd640 1 monclient: found mon.noname-c 2024-03-25T17:56:27.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9efaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:27.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9efaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T17:56:27.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9efaffd640 1 -- 172.21.15.169:0/3728825764 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc070a60 con 0x7f9efc070210 2024-03-25T17:56:27.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9efaffd640 10 monclient: _finish_auth 0 2024-03-25T17:56:27.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9efaffd640 10 monclient: _check_auth_tickets 2024-03-25T17:56:27.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9efaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:27.829 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9efaffd640 1 -- 172.21.15.169:0/3728825764 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9ef0001680 con 0x7f9efc070210 2024-03-25T17:56:27.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9f025b5640 5 monclient: authenticate success, global_id 4123 2024-03-25T17:56:27.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9f025b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:27.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9efaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:27.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9efaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:56:27.830 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9efaffd640 10 monclient: dump: 2024-03-25T17:56:27.830 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:27.830 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:27.830 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.830 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.830 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:27.830 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:27.830 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:27.830 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:27.830 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:27.831 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:27.831 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.828+0000 7f9f025b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:27.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.830+0000 7f9efaffd640 1 -- 172.21.15.169:0/3728825764 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ef00017e0 con 0x7f9efc070210 2024-03-25T17:56:27.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.830+0000 7f9efaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:27.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.830+0000 7f9efaffd640 1 -- 172.21.15.169:0/3728825764 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9ef0001b70 con 0x7f9efc070210 2024-03-25T17:56:27.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.830+0000 7f9f025b5640 10 monclient: get_monmap_and_config success 2024-03-25T17:56:27.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.830+0000 7f9f025b5640 4 set_mon_vals no callback set 2024-03-25T17:56:27.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.830+0000 7f9efaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:27.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.830+0000 7f9efaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:56:27.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.830+0000 7f9efaffd640 10 monclient: dump: 2024-03-25T17:56:27.835 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:27.835 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:27.835 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.835 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.835 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:27.835 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:27.835 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:27.835 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:27.835 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:27.835 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:27.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.831+0000 7f9f025b5640 10 monclient: shutdown 2024-03-25T17:56:27.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.831+0000 7f9f025b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:27.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.831+0000 7f9f025b5640 1 -- 172.21.15.169:0/3728825764 >> v1:172.21.15.169:6789/0 conn(0x7f9efc070210 legacy=0x7f9efc140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:27.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.831+0000 7f9f025b5640 1 -- 172.21.15.169:0/3728825764 shutdown_connections 2024-03-25T17:56:27.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.831+0000 7f9f025b5640 1 -- 172.21.15.169:0/3728825764 >> 172.21.15.169:0/3728825764 conn(0x7f9efc0b5cd0 msgr2=0x7f9efc0b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:27.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.831+0000 7f9f025b5640 1 -- 172.21.15.169:0/3728825764 shutdown_connections 2024-03-25T17:56:27.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.831+0000 7f9f025b5640 1 -- 172.21.15.169:0/3728825764 wait complete. 2024-03-25T17:56:27.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.831+0000 7f9f025b5640 10 monclient: build_initial_monmap 2024-03-25T17:56:27.836 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.831+0000 7f9f025b5640 10 monclient: monmap: 2024-03-25T17:56:27.836 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T17:56:27.836 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:56:27.836 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-25T17:56:27.836 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-25T17:56:27.836 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:27.836 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-25T17:56:27.836 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:27.836 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:56:27.836 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:56:27.837 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:27.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.832+0000 7f9f025b5640 1 librados: starting msgr at 2024-03-25T17:56:27.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.832+0000 7f9f025b5640 1 librados: starting objecter 2024-03-25T17:56:27.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.832+0000 7f9f025b5640 1 Processor -- start 2024-03-25T17:56:27.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.832+0000 7f9f025b5640 1 -- start start 2024-03-25T17:56:27.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.832+0000 7f9f025b5640 1 librados: setting wanted keys 2024-03-25T17:56:27.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.832+0000 7f9f025b5640 1 librados: calling monclient init 2024-03-25T17:56:27.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.832+0000 7f9f025b5640 10 monclient: init 2024-03-25T17:56:27.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.833+0000 7f9f025b5640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:27.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.833+0000 7f9f025b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:56:27.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.833+0000 7f9f025b5640 10 monclient(hunting): picked mon.noname-a con 0x7f9efc070210 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:27.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.833+0000 7f9f025b5640 10 monclient(hunting): picked mon.noname-b con 0x7f9efc0705f0 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:27.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.833+0000 7f9f025b5640 10 monclient(hunting): picked mon.noname-c con 0x7f9efc0b06d0 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:27.837 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.833+0000 7f9f025b5640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc1d7c70 con 0x7f9efc070210 2024-03-25T17:56:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.833+0000 7f9f025b5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc1db9e0 con 0x7f9efc0705f0 2024-03-25T17:56:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.833+0000 7f9f025b5640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc144470 con 0x7f9efc0b06d0 2024-03-25T17:56:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.833+0000 7f9f025b5640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.833+0000 7f9f025b5640 10 monclient(hunting): authenticate will time out at 6903.118638s 2024-03-25T17:56:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.833+0000 7f9efb7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f9efc0705f0 0x7f9efc1db2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:32990/0 (socket says 172.21.15.169:32990) 2024-03-25T17:56:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.833+0000 7f9efbfff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f9efc070210 0x7f9efc1dabc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.169:34668/0 (socket says 172.21.15.169:34668) 2024-03-25T17:56:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.833+0000 7f9efb7fe640 1 -- 172.21.15.169:0/3804306704 learned_addr learned my addr 172.21.15.169:0/3804306704 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T17:56:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.834+0000 7f9ef8ff9640 10 client.?.objecter ms_handle_connect 0x7f9efc070210 2024-03-25T17:56:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.834+0000 7f9ef8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.834+0000 7f9ef8ff9640 10 client.?.objecter ms_handle_connect 0x7f9efc0705f0 2024-03-25T17:56:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.834+0000 7f9ef8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.834+0000 7f9ef8ff9640 10 client.?.objecter ms_handle_connect 0x7f9efc0b06d0 2024-03-25T17:56:27.838 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.834+0000 7f9ef8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:27.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.835+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049470090 0 0) 0x7f9efc144470 con 0x7f9efc0b06d0 2024-03-25T17:56:27.839 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.835+0000 7f9ef8ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:56:27.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.835+0000 7f9ef8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:27.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.835+0000 7f9ef8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:27.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.835+0000 7f9ef8ff9640 10 monclient(hunting): my global_id is 4127 2024-03-25T17:56:27.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.835+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee8004eb0 con 0x7f9efc0b06d0 2024-03-25T17:56:27.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.835+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078644488 0 0) 0x7f9efc1db9e0 con 0x7f9efc0705f0 2024-03-25T17:56:27.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.835+0000 7f9ef8ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:56:27.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.835+0000 7f9ef8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:27.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.835+0000 7f9ef8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:27.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.835+0000 7f9ef8ff9640 10 monclient(hunting): my global_id is 4126 2024-03-25T17:56:27.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.835+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9efc144470 con 0x7f9efc0705f0 2024-03-25T17:56:27.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.836+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369155965 0 0) 0x7f9efc1d7c70 con 0x7f9efc070210 2024-03-25T17:56:27.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.836+0000 7f9ef8ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:56:27.840 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.836+0000 7f9ef8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:27.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.836+0000 7f9ef8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:27.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.836+0000 7f9ef8ff9640 10 monclient(hunting): my global_id is 4125 2024-03-25T17:56:27.841 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.836+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9efc1db9e0 con 0x7f9efc070210 2024-03-25T17:56:27.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.840+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 401412342 0 0) 0x7f9efc1db9e0 con 0x7f9efc070210 2024-03-25T17:56:27.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.840+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9efc1d7c70 con 0x7f9efc070210 2024-03-25T17:56:27.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.841+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3270444669 0 0) 0x7f9efc144470 con 0x7f9efc0705f0 2024-03-25T17:56:27.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.841+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9efc1db9e0 con 0x7f9efc0705f0 2024-03-25T17:56:27.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.841+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9eec001890 con 0x7f9efc0705f0 2024-03-25T17:56:27.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.841+0000 7f9ef8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:27.842 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.841+0000 7f9ef8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:56:27.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.841+0000 7f9ef8ff9640 10 monclient(hunting): dump: 2024-03-25T17:56:27.843 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:27.843 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:27.843 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.843 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.843 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:27.843 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:27.843 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:27.843 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:27.843 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:27.843 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:27.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.841+0000 7f9ef8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:56:27.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.841+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9ef00012a0 con 0x7f9efc070210 2024-03-25T17:56:27.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.841+0000 7f9ef8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:27.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.841+0000 7f9ef8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:27.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.841+0000 7f9ef8ff9640 10 monclient(hunting): dump: 2024-03-25T17:56:27.844 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:27.844 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:27.844 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.844 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.844 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:27.844 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:27.844 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:27.844 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:27.844 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:27.844 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:27.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.842+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 773457781 0 0) 0x7f9ee8004eb0 con 0x7f9efc0b06d0 2024-03-25T17:56:27.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.842+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9efc144470 con 0x7f9efc0b06d0 2024-03-25T17:56:27.844 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.842+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9ef4001c70 con 0x7f9efc0b06d0 2024-03-25T17:56:27.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.842+0000 7f9ef8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:27.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.842+0000 7f9ef8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:56:27.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.842+0000 7f9ef8ff9640 10 monclient(hunting): dump: 2024-03-25T17:56:27.845 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:27.845 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:27.845 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.845 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.845 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:27.845 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:27.845 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:27.845 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:27.845 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:27.845 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:27.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3691835225 0 0) 0x7f9efc1db9e0 con 0x7f9efc0705f0 2024-03-25T17:56:27.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 >> v1:172.21.15.18:6790/0 conn(0x7f9efc0b06d0 legacy=0x7f9efc1e2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:27.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 >> v1:172.21.15.18:6789/0 conn(0x7f9efc070210 legacy=0x7f9efc1dabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:27.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9ef8ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:56:27.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9ef8ff9640 1 monclient: found mon.b 2024-03-25T17:56:27.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9ef8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:27.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9ef8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:27.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc139560 con 0x7f9efc0705f0 2024-03-25T17:56:27.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9ef8ff9640 10 monclient: _finish_auth 0 2024-03-25T17:56:27.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9ef8ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:56:27.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9ef8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:27.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9f025b5640 5 monclient: authenticate success, global_id 4126 2024-03-25T17:56:27.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9f025b5640 10 monclient: _renew_subs 2024-03-25T17:56:27.847 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9f025b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:27.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9f025b5640 1 -- 172.21.15.169:0/3804306704 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9efc13d4e0 con 0x7f9efc0705f0 2024-03-25T17:56:27.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9f025b5640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:56:27.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9f025b5640 10 monclient: _renew_subs 2024-03-25T17:56:27.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9f025b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:27.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9f025b5640 1 -- 172.21.15.169:0/3804306704 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9efc1d78a0 con 0x7f9efc0705f0 2024-03-25T17:56:27.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.844+0000 7f9f025b5640 1 librados: init done 2024-03-25T17:56:27.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.846+0000 7f9f025b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:27.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.846+0000 7f9f025b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:27.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.846+0000 7f9f025b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:27.848 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.846+0000 7f9f025b5640 1 -- 172.21.15.169:0/3804306704 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9efc139380 con 0x7f9efc0705f0 2024-03-25T17:56:27.849 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:27.847+0000 7fc44ee54040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T17:56:27.849 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:27.848+0000 7fb497bf1040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T17:56:27.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.869+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9eec003610 con 0x7f9efc0705f0 2024-03-25T17:56:27.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.869+0000 7f9ef8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:27.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.869+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9eec001d80 con 0x7f9efc0705f0 2024-03-25T17:56:27.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.869+0000 7f9ef8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:27.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.869+0000 7f9ef8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:56:27.869 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.869+0000 7f9ef8ff9640 10 monclient: dump: 2024-03-25T17:56:27.869 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:27.869 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:27.870 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.870 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:27.870 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:27.870 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:27.870 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:27.870 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:27.870 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:27.870 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:27.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.869+0000 7f9efa7fc640 4 set_mon_vals no callback set 2024-03-25T17:56:27.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.869+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9eec003e70 con 0x7f9efc0705f0 2024-03-25T17:56:27.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.869+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2310944780 0 0) 0x7f9eec0063e0 con 0x7f9efc0705f0 2024-03-25T17:56:27.870 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.869+0000 7f9ef8ff9640 10 client.4126.objecter ms_dispatch 0x7f9efc000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T17:56:27.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.869+0000 7f9ef8ff9640 3 client.4126.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T17:56:27.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.869+0000 7f9ef8ff9640 3 client.4126.objecter handle_osd_map decoding full epoch 3 2024-03-25T17:56:27.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:27.869+0000 7f9ef8ff9640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-25T17:56:27.896 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:27.896+0000 7fc44ee54040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T17:56:27.896 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:27.897+0000 7fb497bf1040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T17:56:27.951 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:27.950+0000 7fc44ee54040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T17:56:27.951 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:27.951+0000 7fb497bf1040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T17:56:27.999 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:27.999+0000 7fc44ee54040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T17:56:28.000 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:28.000+0000 7fb497bf1040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T17:56:28.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.038+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9eec004120 con 0x7f9efc0705f0 2024-03-25T17:56:28.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.038+0000 7f9ef8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:28.039 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.038+0000 7f9ef8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:56:28.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.132+0000 7f9f025b5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2f60b59e-0fb3-4ad1-a470-9eec2498bb28", "id": 1}] 2024-03-25T17:56:28.132 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.132+0000 7f9f025b5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2f60b59e-0fb3-4ad1-a470-9eec2498bb28", "id": 1}] 2024-03-25T17:56:28.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.132+0000 7f9f025b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:28.133 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.132+0000 7f9f025b5640 1 -- 172.21.15.169:0/3804306704 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd new", "uuid": "2f60b59e-0fb3-4ad1-a470-9eec2498bb28", "id": 1} v 0) v1 -- 0x7f9efc04f730 con 0x7f9efc0705f0 2024-03-25T17:56:28.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.251+0000 7f9ef8ff9640 1 -- 172.21.15.169:0/3804306704 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2f60b59e-0fb3-4ad1-a470-9eec2498bb28", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3601518640 0 940483276) 0x7f9eec0057c0 con 0x7f9efc0705f0 2024-03-25T17:56:28.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.251+0000 7f9ef8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2f60b59e-0fb3-4ad1-a470-9eec2498bb28", "id": 1}] 2024-03-25T17:56:28.251 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.251+0000 7f9ef8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:56:28.251 INFO:teuthology.orchestra.run.smithi169.stdout:1 2024-03-25T17:56:28.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.251+0000 7f9f025b5640 10 librados: watch_flush enter 2024-03-25T17:56:28.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.252+0000 7f9f025b5640 10 librados: watch_flush exit 2024-03-25T17:56:28.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.252+0000 7f9f025b5640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-25T17:56:28.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.252+0000 7f9f025b5640 10 client.4126.objecter successfully canceled tick 2024-03-25T17:56:28.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.252+0000 7f9f025b5640 10 monclient: shutdown 2024-03-25T17:56:28.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.252+0000 7f9f025b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:28.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.252+0000 7f9f025b5640 1 -- 172.21.15.169:0/3804306704 >> v1:172.21.15.169:6789/0 conn(0x7f9efc0705f0 legacy=0x7f9efc1db2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:28.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.252+0000 7f9f025b5640 1 -- 172.21.15.169:0/3804306704 shutdown_connections 2024-03-25T17:56:28.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.252+0000 7f9f025b5640 1 -- 172.21.15.169:0/3804306704 >> 172.21.15.169:0/3804306704 conn(0x7f9efc0b5cd0 msgr2=0x7f9efc0afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:28.253 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.252+0000 7f9f025b5640 1 -- 172.21.15.169:0/3804306704 shutdown_connections 2024-03-25T17:56:28.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.252+0000 7f9f025b5640 1 -- 172.21.15.169:0/3804306704 wait complete. 2024-03-25T17:56:28.254 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.252+0000 7f9f025b5640 1 librados: shutdown 2024-03-25T17:56:28.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 87509107-33d5-45a2-84a2-48a1d184e4b6 2 2024-03-25T17:56:28.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.364+0000 7f0d10fef640 10 monclient: get_monmap_and_config 2024-03-25T17:56:28.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.364+0000 7f0d10fef640 10 monclient: build_initial_monmap 2024-03-25T17:56:28.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.364+0000 7f0d10fef640 10 monclient: monmap: 2024-03-25T17:56:28.364 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T17:56:28.364 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:28.364 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:56:28.365141+0000 2024-03-25T17:56:28.364 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:56:28.365141+0000 2024-03-25T17:56:28.364 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:28.365 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:28.365 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:28.365 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:56:28.365 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:56:28.365 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:28.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.364+0000 7f0d10fef640 1 Processor -- start 2024-03-25T17:56:28.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.364+0000 7f0d10fef640 1 -- start start 2024-03-25T17:56:28.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.364+0000 7f0d10fef640 10 monclient: init 2024-03-25T17:56:28.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.365+0000 7f0d10fef640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:28.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.365+0000 7f0d10fef640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:56:28.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.365+0000 7f0d10fef640 10 monclient(hunting): picked mon.noname-c con 0x7f0d0c0b4700 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:28.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.365+0000 7f0d10fef640 10 monclient(hunting): picked mon.noname-b con 0x7f0d0c142a20 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:28.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.365+0000 7f0d10fef640 10 monclient(hunting): picked mon.noname-a con 0x7f0d0c0a96b0 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:28.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.365+0000 7f0d10fef640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c0a9d60 con 0x7f0d0c0a96b0 2024-03-25T17:56:28.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.365+0000 7f0d10fef640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c13ff10 con 0x7f0d0c0b4700 2024-03-25T17:56:28.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.365+0000 7f0d10fef640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c0aa1f0 con 0x7f0d0c142a20 2024-03-25T17:56:28.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.365+0000 7f0d10fef640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:28.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.365+0000 7f0d10fef640 10 monclient(hunting): authenticate will time out at 6903.650900s 2024-03-25T17:56:28.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.366+0000 7f0d0ad9d640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f0d0c0a96b0 0x7f0d0c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.169:34676/0 (socket says 172.21.15.169:34676) 2024-03-25T17:56:28.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.366+0000 7f0d0ad9d640 1 -- 172.21.15.169:0/3389919484 learned_addr learned my addr 172.21.15.169:0/3389919484 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T17:56:28.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 1 -- 172.21.15.169:0/3389919484 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668540235 0 0) 0x7f0d0c0aa1f0 con 0x7f0d0c142a20 2024-03-25T17:56:28.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 10 monclient(hunting): _negotiate 2024-03-25T17:56:28.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:28.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:28.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 10 monclient(hunting): my global_id is 4130 2024-03-25T17:56:28.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 1 -- 172.21.15.169:0/3389919484 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf4003520 con 0x7f0d0c142a20 2024-03-25T17:56:28.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 1 -- 172.21.15.169:0/3389919484 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33159242 0 0) 0x7f0d0c13ff10 con 0x7f0d0c0b4700 2024-03-25T17:56:28.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 10 monclient(hunting): _negotiate 2024-03-25T17:56:28.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:28.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:28.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 10 monclient(hunting): my global_id is 4129 2024-03-25T17:56:28.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 1 -- 172.21.15.169:0/3389919484 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d0c0aa1f0 con 0x7f0d0c0b4700 2024-03-25T17:56:28.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 1 -- 172.21.15.169:0/3389919484 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673251699 0 0) 0x7f0d0c0a9d60 con 0x7f0d0c0a96b0 2024-03-25T17:56:28.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 10 monclient(hunting): _negotiate 2024-03-25T17:56:28.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:28.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:28.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 10 monclient(hunting): my global_id is 4128 2024-03-25T17:56:28.370 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.368+0000 7f0d0959a640 1 -- 172.21.15.169:0/3389919484 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d0c13ff10 con 0x7f0d0c0a96b0 2024-03-25T17:56:28.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.372+0000 7f0d0959a640 1 -- 172.21.15.169:0/3389919484 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 336623134 0 0) 0x7f0d0c13ff10 con 0x7f0d0c0a96b0 2024-03-25T17:56:28.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.372+0000 7f0d0959a640 1 -- 172.21.15.169:0/3389919484 >> v1:172.21.15.18:6790/0 conn(0x7f0d0c142a20 legacy=0x7f0d0c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:28.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.372+0000 7f0d0959a640 1 -- 172.21.15.169:0/3389919484 >> v1:172.21.15.169:6789/0 conn(0x7f0d0c0b4700 legacy=0x7f0d0c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:28.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.372+0000 7f0d0959a640 10 monclient: _finish_hunting 0 2024-03-25T17:56:28.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.372+0000 7f0d0959a640 1 monclient: found mon.noname-a 2024-03-25T17:56:28.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.372+0000 7f0d0959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:28.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.372+0000 7f0d0959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T17:56:28.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.372+0000 7f0d0959a640 1 -- 172.21.15.169:0/3389919484 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c0b4ae0 con 0x7f0d0c0a96b0 2024-03-25T17:56:28.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.372+0000 7f0d0959a640 10 monclient: _finish_auth 0 2024-03-25T17:56:28.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.372+0000 7f0d0959a640 10 monclient: _check_auth_tickets 2024-03-25T17:56:28.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.372+0000 7f0d0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:28.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.372+0000 7f0d0959a640 1 -- 172.21.15.169:0/3389919484 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0d04001930 con 0x7f0d0c0a96b0 2024-03-25T17:56:28.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.372+0000 7f0d0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:28.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.372+0000 7f0d0959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:56:28.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.372+0000 7f0d0959a640 10 monclient: dump: 2024-03-25T17:56:28.374 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:28.374 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:28.374 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.374 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.374 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:28.374 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:28.374 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:28.374 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:28.374 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:28.374 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:28.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.372+0000 7f0d10fef640 5 monclient: authenticate success, global_id 4128 2024-03-25T17:56:28.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.372+0000 7f0d10fef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:28.377 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.374+0000 7f0d0959a640 1 -- 172.21.15.169:0/3389919484 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d04001a90 con 0x7f0d0c0a96b0 2024-03-25T17:56:28.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.374+0000 7f0d0959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:28.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.374+0000 7f0d10fef640 10 monclient: get_monmap_and_config success 2024-03-25T17:56:28.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.374+0000 7f0d10fef640 4 set_mon_vals no callback set 2024-03-25T17:56:28.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.374+0000 7f0d0959a640 1 -- 172.21.15.169:0/3389919484 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0d04001ee0 con 0x7f0d0c0a96b0 2024-03-25T17:56:28.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.374+0000 7f0d0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:28.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.374+0000 7f0d0959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:28.378 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.374+0000 7f0d0959a640 10 monclient: dump: 2024-03-25T17:56:28.378 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:28.378 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:28.378 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.378 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.378 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:28.379 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:28.379 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:28.379 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:28.379 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:28.379 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:28.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.374+0000 7f0d10fef640 10 monclient: shutdown 2024-03-25T17:56:28.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.374+0000 7f0d10fef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:28.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.374+0000 7f0d10fef640 1 -- 172.21.15.169:0/3389919484 >> v1:172.21.15.18:6789/0 conn(0x7f0d0c0a96b0 legacy=0x7f0d0c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:28.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.374+0000 7f0d10fef640 1 -- 172.21.15.169:0/3389919484 shutdown_connections 2024-03-25T17:56:28.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.374+0000 7f0d10fef640 1 -- 172.21.15.169:0/3389919484 >> 172.21.15.169:0/3389919484 conn(0x7f0d0c0668e0 msgr2=0x7f0d0c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:28.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.374+0000 7f0d10fef640 1 -- 172.21.15.169:0/3389919484 shutdown_connections 2024-03-25T17:56:28.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.374+0000 7f0d10fef640 1 -- 172.21.15.169:0/3389919484 wait complete. 2024-03-25T17:56:28.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.375+0000 7f0d10fef640 10 monclient: build_initial_monmap 2024-03-25T17:56:28.379 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.375+0000 7f0d10fef640 10 monclient: monmap: 2024-03-25T17:56:28.380 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T17:56:28.380 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:56:28.380 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-25T17:56:28.380 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-25T17:56:28.380 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:28.380 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-25T17:56:28.380 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:28.380 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:56:28.380 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:56:28.380 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:28.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.376+0000 7f0d10fef640 1 librados: starting msgr at 2024-03-25T17:56:28.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.376+0000 7f0d10fef640 1 librados: starting objecter 2024-03-25T17:56:28.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.376+0000 7f0d10fef640 1 Processor -- start 2024-03-25T17:56:28.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.376+0000 7f0d10fef640 1 -- start start 2024-03-25T17:56:28.380 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.376+0000 7f0d10fef640 1 librados: setting wanted keys 2024-03-25T17:56:28.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.376+0000 7f0d10fef640 1 librados: calling monclient init 2024-03-25T17:56:28.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.376+0000 7f0d10fef640 10 monclient: init 2024-03-25T17:56:28.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.376+0000 7f0d10fef640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:28.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.376+0000 7f0d10fef640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:56:28.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.376+0000 7f0d10fef640 10 monclient(hunting): picked mon.noname-a con 0x7f0d0c0a96b0 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:28.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.376+0000 7f0d10fef640 10 monclient(hunting): picked mon.noname-b con 0x7f0d0c0b4700 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:28.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.376+0000 7f0d10fef640 10 monclient(hunting): picked mon.noname-c con 0x7f0d0c142a20 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:28.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.376+0000 7f0d10fef640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c136680 con 0x7f0d0c0a96b0 2024-03-25T17:56:28.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.376+0000 7f0d10fef640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c13a210 con 0x7f0d0c0b4700 2024-03-25T17:56:28.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.376+0000 7f0d10fef640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c1d83e0 con 0x7f0d0c142a20 2024-03-25T17:56:28.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.376+0000 7f0d10fef640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:28.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.376+0000 7f0d10fef640 10 monclient(hunting): authenticate will time out at 6903.662089s 2024-03-25T17:56:28.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.377+0000 7f0d0a59c640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f0d0c0a96b0 0x7f0d0c1d7cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.169:34684/0 (socket says 172.21.15.169:34684) 2024-03-25T17:56:28.381 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.377+0000 7f0d0a59c640 1 -- 172.21.15.169:0/1361082729 learned_addr learned my addr 172.21.15.169:0/1361082729 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T17:56:28.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.377+0000 7f0cfaffd640 10 client.?.objecter ms_handle_connect 0x7f0d0c0a96b0 2024-03-25T17:56:28.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.377+0000 7f0cfaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:28.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.377+0000 7f0cfaffd640 10 client.?.objecter ms_handle_connect 0x7f0d0c0b4700 2024-03-25T17:56:28.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.377+0000 7f0cfaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:28.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.377+0000 7f0cfaffd640 10 client.?.objecter ms_handle_connect 0x7f0d0c142a20 2024-03-25T17:56:28.382 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.377+0000 7f0cfaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:28.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280101490 0 0) 0x7f0d0c1d83e0 con 0x7f0d0c142a20 2024-03-25T17:56:28.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:56:28.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:28.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:28.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 10 monclient(hunting): my global_id is 4133 2024-03-25T17:56:28.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cec004eb0 con 0x7f0d0c142a20 2024-03-25T17:56:28.383 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63797720 0 0) 0x7f0d0c136680 con 0x7f0d0c0a96b0 2024-03-25T17:56:28.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:56:28.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:28.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:28.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 10 monclient(hunting): my global_id is 4131 2024-03-25T17:56:28.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d0c1d83e0 con 0x7f0d0c0a96b0 2024-03-25T17:56:28.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588858024 0 0) 0x7f0d0c13a210 con 0x7f0d0c0b4700 2024-03-25T17:56:28.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:56:28.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:28.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:28.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 10 monclient(hunting): my global_id is 4132 2024-03-25T17:56:28.384 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.379+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d0c136680 con 0x7f0d0c0b4700 2024-03-25T17:56:28.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.384+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 537473451 0 0) 0x7f0d0c1d83e0 con 0x7f0d0c0a96b0 2024-03-25T17:56:28.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.384+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d0c13a210 con 0x7f0d0c0a96b0 2024-03-25T17:56:28.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.384+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0cfc0019b0 con 0x7f0d0c0a96b0 2024-03-25T17:56:28.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.384+0000 7f0cfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:28.385 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.385+0000 7f0cfaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:56:28.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.385+0000 7f0cfaffd640 10 monclient(hunting): dump: 2024-03-25T17:56:28.386 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:28.386 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:28.386 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.386 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.386 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:28.386 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:28.386 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:28.386 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:28.386 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:28.386 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:28.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.385+0000 7f0cfaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:56:28.386 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.385+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3217311807 0 0) 0x7f0cec004eb0 con 0x7f0d0c142a20 2024-03-25T17:56:28.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.385+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d0c1d83e0 con 0x7f0d0c142a20 2024-03-25T17:56:28.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.385+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0d04001da0 con 0x7f0d0c142a20 2024-03-25T17:56:28.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.385+0000 7f0cfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:28.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.385+0000 7f0cfaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:56:28.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.385+0000 7f0cfaffd640 10 monclient(hunting): dump: 2024-03-25T17:56:28.387 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:28.387 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:28.387 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.387 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.387 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:28.387 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:28.387 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:28.387 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:28.387 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:28.388 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:28.389 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.387+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1074410936 0 0) 0x7f0d0c136680 con 0x7f0d0c0b4700 2024-03-25T17:56:28.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.387+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cec004eb0 con 0x7f0d0c0b4700 2024-03-25T17:56:28.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.387+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0d00001750 con 0x7f0d0c0b4700 2024-03-25T17:56:28.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.387+0000 7f0cfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:28.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.387+0000 7f0cfaffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:56:28.390 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.387+0000 7f0cfaffd640 10 monclient(hunting): dump: 2024-03-25T17:56:28.390 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:28.390 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:28.390 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.390 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.390 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:28.390 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:28.390 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:28.391 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:28.391 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:28.391 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:28.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 958298003 0 0) 0x7f0d0c1d83e0 con 0x7f0d0c142a20 2024-03-25T17:56:28.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 >> v1:172.21.15.169:6789/0 conn(0x7f0d0c0b4700 legacy=0x7f0d0c1d47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:28.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 >> v1:172.21.15.18:6789/0 conn(0x7f0d0c0a96b0 legacy=0x7f0d0c1d7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:28.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0cfaffd640 10 monclient: _finish_hunting 0 2024-03-25T17:56:28.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0cfaffd640 1 monclient: found mon.c 2024-03-25T17:56:28.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0cfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:28.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0cfaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:56:28.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c1d74d0 con 0x7f0d0c142a20 2024-03-25T17:56:28.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0cfaffd640 10 monclient: _finish_auth 0 2024-03-25T17:56:28.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0cfaffd640 10 monclient: _check_auth_tickets 2024-03-25T17:56:28.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0cfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:28.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0d10fef640 5 monclient: authenticate success, global_id 4133 2024-03-25T17:56:28.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0d10fef640 10 monclient: _renew_subs 2024-03-25T17:56:28.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0d10fef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:56:28.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0d10fef640 1 -- 172.21.15.169:0/1361082729 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0c1d77b0 con 0x7f0d0c142a20 2024-03-25T17:56:28.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0d10fef640 10 client.4133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:56:28.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0d10fef640 10 monclient: _renew_subs 2024-03-25T17:56:28.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0d10fef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:56:28.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0d10fef640 1 -- 172.21.15.169:0/1361082729 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0c138c30 con 0x7f0d0c142a20 2024-03-25T17:56:28.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.388+0000 7f0d10fef640 1 librados: init done 2024-03-25T17:56:28.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.389+0000 7f0d10fef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:28.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.389+0000 7f0d10fef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:28.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.389+0000 7f0d10fef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:56:28.392 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.389+0000 7f0d10fef640 1 -- 172.21.15.169:0/1361082729 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0c04f730 con 0x7f0d0c142a20 2024-03-25T17:56:28.406 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:28.406+0000 7fc44ee54040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T17:56:28.424 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:28.424+0000 7fb497bf1040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T17:56:28.549 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:28.548+0000 7fc44ee54040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T17:56:28.562 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:28.562+0000 7fb497bf1040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T17:56:28.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.612+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d04001f40 con 0x7f0d0c142a20 2024-03-25T17:56:28.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.612+0000 7f0cfaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:28.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.612+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0d04004fb0 con 0x7f0d0c142a20 2024-03-25T17:56:28.612 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.612+0000 7f0cfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:28.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.612+0000 7f0cfaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:56:28.613 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.612+0000 7f0cfaffd640 10 monclient: dump: 2024-03-25T17:56:28.613 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:28.613 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:28.613 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.613 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.613 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:28.613 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:28.613 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:28.613 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:28.613 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:28.613 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:28.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.612+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0d040054a0 con 0x7f0d0c142a20 2024-03-25T17:56:28.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.612+0000 7f0d08d99640 4 set_mon_vals no callback set 2024-03-25T17:56:28.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.614+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1133845072 0 0) 0x7f0d040069d0 con 0x7f0d0c142a20 2024-03-25T17:56:28.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.614+0000 7f0cfaffd640 10 client.4133.objecter ms_dispatch 0x7f0d0c000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T17:56:28.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.614+0000 7f0cfaffd640 3 client.4133.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T17:56:28.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.614+0000 7f0cfaffd640 3 client.4133.objecter handle_osd_map decoding full epoch 4 2024-03-25T17:56:28.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.614+0000 7f0cfaffd640 20 client.4133.objecter dump_active .. 0 homeless 2024-03-25T17:56:28.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.615+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0d040065b0 con 0x7f0d0c142a20 2024-03-25T17:56:28.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.615+0000 7f0cfaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:28.615 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.615+0000 7f0cfaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:56:28.685 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:28.684+0000 7fc44ee54040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T17:56:28.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.687+0000 7f0d10fef640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "87509107-33d5-45a2-84a2-48a1d184e4b6", "id": 2}] 2024-03-25T17:56:28.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.687+0000 7f0d10fef640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "87509107-33d5-45a2-84a2-48a1d184e4b6", "id": 2}] 2024-03-25T17:56:28.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.687+0000 7f0d10fef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:56:28.687 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.687+0000 7f0d10fef640 1 -- 172.21.15.169:0/1361082729 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd new", "uuid": "87509107-33d5-45a2-84a2-48a1d184e4b6", "id": 2} v 0) v1 -- 0x7f0d0c0b6000 con 0x7f0d0c142a20 2024-03-25T17:56:28.690 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:28.690+0000 7fb497bf1040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T17:56:28.733 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:28.733+0000 7fc44ee54040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T17:56:28.736 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:28.736+0000 7fb497bf1040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T17:56:28.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.825+0000 7f0cfaffd640 1 -- 172.21.15.169:0/1361082729 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "87509107-33d5-45a2-84a2-48a1d184e4b6", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1698908202 0 216599125) 0x7f0d040058e0 con 0x7f0d0c142a20 2024-03-25T17:56:28.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.825+0000 7f0cfaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "87509107-33d5-45a2-84a2-48a1d184e4b6", "id": 2}] 2024-03-25T17:56:28.825 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.825+0000 7f0cfaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:56:28.826 INFO:teuthology.orchestra.run.smithi169.stdout:2 2024-03-25T17:56:28.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.826+0000 7f0d10fef640 10 librados: watch_flush enter 2024-03-25T17:56:28.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.826+0000 7f0d10fef640 10 librados: watch_flush exit 2024-03-25T17:56:28.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.826+0000 7f0d10fef640 20 client.4133.objecter shutdown clearing up homeless session... 2024-03-25T17:56:28.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.826+0000 7f0d10fef640 10 client.4133.objecter successfully canceled tick 2024-03-25T17:56:28.827 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.826+0000 7f0d10fef640 10 monclient: shutdown 2024-03-25T17:56:28.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.826+0000 7f0d10fef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:28.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.826+0000 7f0d10fef640 1 -- 172.21.15.169:0/1361082729 >> v1:172.21.15.18:6790/0 conn(0x7f0d0c142a20 legacy=0x7f0d0c138520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:28.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.826+0000 7f0d10fef640 1 -- 172.21.15.169:0/1361082729 shutdown_connections 2024-03-25T17:56:28.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.826+0000 7f0d10fef640 1 -- 172.21.15.169:0/1361082729 >> 172.21.15.169:0/1361082729 conn(0x7f0d0c0668e0 msgr2=0x7f0d0c142220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:28.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.826+0000 7f0d10fef640 1 -- 172.21.15.169:0/1361082729 shutdown_connections 2024-03-25T17:56:28.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.826+0000 7f0d10fef640 1 -- 172.21.15.169:0/1361082729 wait complete. 2024-03-25T17:56:28.828 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.826+0000 7f0d10fef640 1 librados: shutdown 2024-03-25T17:56:28.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new d91f737b-db6a-4b15-ad8c-b8a546d3536c 3 2024-03-25T17:56:28.856 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:28.856+0000 7fb497bf1040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T17:56:28.857 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:28.856+0000 7fc44ee54040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T17:56:28.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.936+0000 7f041e214640 10 monclient: get_monmap_and_config 2024-03-25T17:56:28.936 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.936+0000 7f041e214640 10 monclient: build_initial_monmap 2024-03-25T17:56:28.937 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.936+0000 7f041e214640 10 monclient: monmap: 2024-03-25T17:56:28.937 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T17:56:28.937 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:28.937 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:56:28.937020+0000 2024-03-25T17:56:28.937 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:56:28.937020+0000 2024-03-25T17:56:28.937 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:28.937 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:28.937 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:28.937 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:56:28.937 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:56:28.937 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:28.937 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.936+0000 7f041e214640 1 Processor -- start 2024-03-25T17:56:28.937 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.936+0000 7f041e214640 1 -- start start 2024-03-25T17:56:28.937 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.936+0000 7f041e214640 10 monclient: init 2024-03-25T17:56:28.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.937+0000 7f041e214640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:28.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.937+0000 7f041e214640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:56:28.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.937+0000 7f041e214640 10 monclient(hunting): picked mon.noname-b con 0x7f04180b4700 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:28.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.937+0000 7f041e214640 10 monclient(hunting): picked mon.noname-a con 0x7f0418142a20 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:28.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.937+0000 7f041e214640 10 monclient(hunting): picked mon.noname-c con 0x7f04180a96b0 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:28.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.937+0000 7f041e214640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04180a9d60 con 0x7f0418142a20 2024-03-25T17:56:28.939 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.937+0000 7f041e214640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041813ff10 con 0x7f04180a96b0 2024-03-25T17:56:28.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.937+0000 7f041e214640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04180aa1f0 con 0x7f04180b4700 2024-03-25T17:56:28.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.937+0000 7f041e214640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:28.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.937+0000 7f041e214640 10 monclient(hunting): authenticate will time out at 6904.222879s 2024-03-25T17:56:28.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.938+0000 7f0416ffd640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f0418142a20 0x7f041813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.169:34690/0 (socket says 172.21.15.169:34690) 2024-03-25T17:56:28.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.938+0000 7f0416ffd640 1 -- 172.21.15.169:0/2278222631 learned_addr learned my addr 172.21.15.169:0/2278222631 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T17:56:28.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 1 -- 172.21.15.169:0/2278222631 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854485850 0 0) 0x7f04180aa1f0 con 0x7f04180b4700 2024-03-25T17:56:28.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 10 monclient(hunting): _negotiate 2024-03-25T17:56:28.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:28.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:28.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 10 monclient(hunting): my global_id is 4136 2024-03-25T17:56:28.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 1 -- 172.21.15.169:0/2278222631 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0400003520 con 0x7f04180b4700 2024-03-25T17:56:28.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 1 -- 172.21.15.169:0/2278222631 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396018255 0 0) 0x7f04180a9d60 con 0x7f0418142a20 2024-03-25T17:56:28.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 10 monclient(hunting): _negotiate 2024-03-25T17:56:28.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:28.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:28.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 10 monclient(hunting): my global_id is 4134 2024-03-25T17:56:28.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 1 -- 172.21.15.169:0/2278222631 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04180aa1f0 con 0x7f0418142a20 2024-03-25T17:56:28.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 1 -- 172.21.15.169:0/2278222631 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74260456 0 0) 0x7f041813ff10 con 0x7f04180a96b0 2024-03-25T17:56:28.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 10 monclient(hunting): _negotiate 2024-03-25T17:56:28.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:28.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:28.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 10 monclient(hunting): my global_id is 4135 2024-03-25T17:56:28.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.940+0000 7f04167fc640 1 -- 172.21.15.169:0/2278222631 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04180a9d60 con 0x7f04180a96b0 2024-03-25T17:56:28.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.943+0000 7f04167fc640 1 -- 172.21.15.169:0/2278222631 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2632756047 0 0) 0x7f04180aa1f0 con 0x7f0418142a20 2024-03-25T17:56:28.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.943+0000 7f04167fc640 1 -- 172.21.15.169:0/2278222631 >> v1:172.21.15.18:6790/0 conn(0x7f04180b4700 legacy=0x7f0418142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:28.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.943+0000 7f04167fc640 1 -- 172.21.15.169:0/2278222631 >> v1:172.21.15.169:6789/0 conn(0x7f04180a96b0 legacy=0x7f041813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:28.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.943+0000 7f04167fc640 10 monclient: _finish_hunting 0 2024-03-25T17:56:28.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.943+0000 7f04167fc640 1 monclient: found mon.noname-a 2024-03-25T17:56:28.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.943+0000 7f04167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:28.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.943+0000 7f04167fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T17:56:28.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.943+0000 7f04167fc640 1 -- 172.21.15.169:0/2278222631 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04180b4ae0 con 0x7f0418142a20 2024-03-25T17:56:28.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.943+0000 7f04167fc640 10 monclient: _finish_auth 0 2024-03-25T17:56:28.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.943+0000 7f04167fc640 10 monclient: _check_auth_tickets 2024-03-25T17:56:28.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.943+0000 7f04167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:28.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.943+0000 7f04167fc640 1 -- 172.21.15.169:0/2278222631 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f04080015f0 con 0x7f0418142a20 2024-03-25T17:56:28.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.944+0000 7f04167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:28.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.944+0000 7f04167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:56:28.945 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.944+0000 7f04167fc640 10 monclient: dump: 2024-03-25T17:56:28.945 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:28.945 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:28.945 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.945 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.945 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:28.946 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:28.946 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:28.946 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:28.946 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:28.946 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:28.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.944+0000 7f041e214640 5 monclient: authenticate success, global_id 4134 2024-03-25T17:56:28.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.944+0000 7f041e214640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:28.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.945+0000 7f04167fc640 1 -- 172.21.15.169:0/2278222631 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0408001750 con 0x7f0418142a20 2024-03-25T17:56:28.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.945+0000 7f04167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:28.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.945+0000 7f04167fc640 1 -- 172.21.15.169:0/2278222631 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0408001b60 con 0x7f0418142a20 2024-03-25T17:56:28.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.945+0000 7f04167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:28.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.945+0000 7f04167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:28.949 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.945+0000 7f04167fc640 10 monclient: dump: 2024-03-25T17:56:28.949 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:28.949 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:28.949 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.949 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.950 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:28.950 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:28.950 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:28.950 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:28.950 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:28.950 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:28.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.945+0000 7f041e214640 10 monclient: get_monmap_and_config success 2024-03-25T17:56:28.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.945+0000 7f041e214640 4 set_mon_vals no callback set 2024-03-25T17:56:28.950 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.946+0000 7f041e214640 10 monclient: shutdown 2024-03-25T17:56:28.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.946+0000 7f041e214640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:28.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.946+0000 7f041e214640 1 -- 172.21.15.169:0/2278222631 >> v1:172.21.15.18:6789/0 conn(0x7f0418142a20 legacy=0x7f041813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:28.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.946+0000 7f041e214640 1 -- 172.21.15.169:0/2278222631 shutdown_connections 2024-03-25T17:56:28.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.946+0000 7f041e214640 1 -- 172.21.15.169:0/2278222631 >> 172.21.15.169:0/2278222631 conn(0x7f04180668e0 msgr2=0x7f0418066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:28.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.946+0000 7f041e214640 1 -- 172.21.15.169:0/2278222631 shutdown_connections 2024-03-25T17:56:28.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.946+0000 7f041e214640 1 -- 172.21.15.169:0/2278222631 wait complete. 2024-03-25T17:56:28.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.947+0000 7f041e214640 10 monclient: build_initial_monmap 2024-03-25T17:56:28.951 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.947+0000 7f041e214640 10 monclient: monmap: 2024-03-25T17:56:28.951 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T17:56:28.951 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:56:28.951 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-25T17:56:28.951 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-25T17:56:28.951 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:28.951 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-25T17:56:28.951 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:28.952 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:56:28.952 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:56:28.952 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:28.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.947+0000 7f041e214640 1 librados: starting msgr at 2024-03-25T17:56:28.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.947+0000 7f041e214640 1 librados: starting objecter 2024-03-25T17:56:28.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.947+0000 7f041e214640 1 Processor -- start 2024-03-25T17:56:28.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.947+0000 7f041e214640 1 -- start start 2024-03-25T17:56:28.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.947+0000 7f041e214640 1 librados: setting wanted keys 2024-03-25T17:56:28.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.947+0000 7f041e214640 1 librados: calling monclient init 2024-03-25T17:56:28.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.947+0000 7f041e214640 10 monclient: init 2024-03-25T17:56:28.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.948+0000 7f041e214640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:28.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.948+0000 7f041e214640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:56:28.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.948+0000 7f041e214640 10 monclient(hunting): picked mon.noname-a con 0x7f04180a96b0 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:28.952 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.948+0000 7f041e214640 10 monclient(hunting): picked mon.noname-b con 0x7f04180b4700 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:28.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.948+0000 7f041e214640 10 monclient(hunting): picked mon.noname-c con 0x7f0418142a20 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:28.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.948+0000 7f041e214640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041813c060 con 0x7f04180a96b0 2024-03-25T17:56:28.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.948+0000 7f041e214640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041813ca90 con 0x7f04180b4700 2024-03-25T17:56:28.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.948+0000 7f041e214640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04180aba90 con 0x7f0418142a20 2024-03-25T17:56:28.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.948+0000 7f041e214640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:28.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.948+0000 7f041e214640 10 monclient(hunting): authenticate will time out at 6904.233902s 2024-03-25T17:56:28.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.948+0000 7f04177fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f04180a96b0 0x7f041813b800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.169:34694/0 (socket says 172.21.15.169:34694) 2024-03-25T17:56:28.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.948+0000 7f04177fe640 1 -- 172.21.15.169:0/3998518633 learned_addr learned my addr 172.21.15.169:0/3998518633 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T17:56:28.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.949+0000 7f03fffff640 10 client.?.objecter ms_handle_connect 0x7f04180a96b0 2024-03-25T17:56:28.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.949+0000 7f03fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:28.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.949+0000 7f03fffff640 10 client.?.objecter ms_handle_connect 0x7f0418142a20 2024-03-25T17:56:28.953 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.949+0000 7f03fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:28.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.951+0000 7f03fffff640 10 client.?.objecter ms_handle_connect 0x7f04180b4700 2024-03-25T17:56:28.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.951+0000 7f03fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:28.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.951+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103401295 0 0) 0x7f041813c060 con 0x7f04180a96b0 2024-03-25T17:56:28.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.951+0000 7f03fffff640 10 monclient(hunting): _negotiate 2024-03-25T17:56:28.955 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.951+0000 7f03fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:28.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.951+0000 7f03fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:28.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.951+0000 7f03fffff640 10 monclient(hunting): my global_id is 4137 2024-03-25T17:56:28.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.951+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0404004eb0 con 0x7f04180a96b0 2024-03-25T17:56:28.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.951+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168298872 0 0) 0x7f04180aba90 con 0x7f0418142a20 2024-03-25T17:56:28.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.951+0000 7f03fffff640 10 monclient(hunting): _negotiate 2024-03-25T17:56:28.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.951+0000 7f03fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:28.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.951+0000 7f03fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:28.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.951+0000 7f03fffff640 10 monclient(hunting): my global_id is 4139 2024-03-25T17:56:28.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.951+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f041813c060 con 0x7f0418142a20 2024-03-25T17:56:28.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.953+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493035353 0 0) 0x7f041813ca90 con 0x7f04180b4700 2024-03-25T17:56:28.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.953+0000 7f03fffff640 10 monclient(hunting): _negotiate 2024-03-25T17:56:28.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.953+0000 7f03fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:28.956 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.953+0000 7f03fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:28.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.953+0000 7f03fffff640 10 monclient(hunting): my global_id is 4138 2024-03-25T17:56:28.957 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.953+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04180aba90 con 0x7f04180b4700 2024-03-25T17:56:28.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.956+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3306552576 0 0) 0x7f0404004eb0 con 0x7f04180a96b0 2024-03-25T17:56:28.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.956+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f041813ca90 con 0x7f04180a96b0 2024-03-25T17:56:28.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.956+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f040c001a10 con 0x7f04180a96b0 2024-03-25T17:56:28.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.956+0000 7f03fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:28.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.956+0000 7f03fffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:56:28.958 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.956+0000 7f03fffff640 10 monclient(hunting): dump: 2024-03-25T17:56:28.958 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:28.958 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:28.958 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.958 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.958 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:28.959 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:28.959 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:28.959 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:28.959 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:28.959 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:28.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.956+0000 7f03fffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:56:28.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.957+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2741755970 0 0) 0x7f041813c060 con 0x7f0418142a20 2024-03-25T17:56:28.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.957+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0404004eb0 con 0x7f0418142a20 2024-03-25T17:56:28.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.957+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0410001b50 con 0x7f0418142a20 2024-03-25T17:56:28.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.957+0000 7f03fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:28.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.957+0000 7f03fffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:56:28.959 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.957+0000 7f03fffff640 10 monclient(hunting): dump: 2024-03-25T17:56:28.959 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:28.959 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:28.960 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.960 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.960 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:28.960 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:28.960 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:28.960 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:28.960 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:28.960 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:28.960 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:28.959+0000 7fc44ee54040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T17:56:28.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 29980739 0 0) 0x7f04180aba90 con 0x7f04180b4700 2024-03-25T17:56:28.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f041813c060 con 0x7f04180b4700 2024-03-25T17:56:28.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0408001a60 con 0x7f04180b4700 2024-03-25T17:56:28.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f03fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:28.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f03fffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:56:28.961 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f03fffff640 10 monclient(hunting): dump: 2024-03-25T17:56:28.962 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:28.962 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:28.962 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.962 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:28.962 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:28.962 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:28.962 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:28.962 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:28.962 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:28.962 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:28.963 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:28.959+0000 7fb497bf1040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T17:56:28.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2418153533 0 0) 0x7f041813ca90 con 0x7f04180a96b0 2024-03-25T17:56:28.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 >> v1:172.21.15.18:6790/0 conn(0x7f0418142a20 legacy=0x7f0418138d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:28.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 >> v1:172.21.15.169:6789/0 conn(0x7f04180b4700 legacy=0x7f0418138650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:28.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f03fffff640 10 monclient: _finish_hunting 0 2024-03-25T17:56:28.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f03fffff640 1 monclient: found mon.a 2024-03-25T17:56:28.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f03fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:28.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f03fffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:28.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041813af40 con 0x7f04180a96b0 2024-03-25T17:56:28.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f03fffff640 10 monclient: _finish_auth 0 2024-03-25T17:56:28.964 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f03fffff640 10 monclient: _check_auth_tickets 2024-03-25T17:56:28.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f03fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:28.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f041e214640 5 monclient: authenticate success, global_id 4137 2024-03-25T17:56:28.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f041e214640 10 monclient: _renew_subs 2024-03-25T17:56:28.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f041e214640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:28.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f041e214640 1 -- 172.21.15.169:0/3998518633 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f041813b1f0 con 0x7f04180a96b0 2024-03-25T17:56:28.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f041e214640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:56:28.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f041e214640 10 monclient: _renew_subs 2024-03-25T17:56:28.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f041e214640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:28.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f041e214640 1 -- 172.21.15.169:0/3998518633 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04181e2e40 con 0x7f04180a96b0 2024-03-25T17:56:28.965 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.959+0000 7f041e214640 1 librados: init done 2024-03-25T17:56:28.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.960+0000 7f041e214640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:28.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.960+0000 7f041e214640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:28.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.960+0000 7f041e214640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:28.966 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:28.960+0000 7f041e214640 1 -- 172.21.15.169:0/3998518633 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f041804f730 con 0x7f04180a96b0 2024-03-25T17:56:29.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.064+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f040c001c50 con 0x7f04180a96b0 2024-03-25T17:56:29.064 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.064+0000 7f03fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:29.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.064+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f040c003a60 con 0x7f04180a96b0 2024-03-25T17:56:29.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.064+0000 7f03fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:29.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.064+0000 7f03fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:29.065 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.064+0000 7f03fffff640 10 monclient: dump: 2024-03-25T17:56:29.065 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:29.065 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:29.065 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.065 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.065 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:29.065 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:29.065 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:29.065 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:29.065 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:29.065 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:29.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.064+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f040c004010 con 0x7f04180a96b0 2024-03-25T17:56:29.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.064+0000 7f0415ffb640 4 set_mon_vals no callback set 2024-03-25T17:56:29.066 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.065+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3995022278 0 0) 0x7f040c006700 con 0x7f04180a96b0 2024-03-25T17:56:29.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.065+0000 7f03fffff640 10 client.4137.objecter ms_dispatch 0x7f0418000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T17:56:29.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.066+0000 7f03fffff640 3 client.4137.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T17:56:29.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.066+0000 7f03fffff640 3 client.4137.objecter handle_osd_map decoding full epoch 5 2024-03-25T17:56:29.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.066+0000 7f03fffff640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-25T17:56:29.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.066+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f040c006270 con 0x7f04180a96b0 2024-03-25T17:56:29.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.066+0000 7f03fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:29.067 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.066+0000 7f03fffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:56:29.116 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:29.116+0000 7fb497bf1040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T17:56:29.121 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:29.121+0000 7fc44ee54040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T17:56:29.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.136+0000 7f041e214640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d91f737b-db6a-4b15-ad8c-b8a546d3536c", "id": 3}] 2024-03-25T17:56:29.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.136+0000 7f041e214640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d91f737b-db6a-4b15-ad8c-b8a546d3536c", "id": 3}] 2024-03-25T17:56:29.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.136+0000 7f041e214640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:29.137 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.136+0000 7f041e214640 1 -- 172.21.15.169:0/3998518633 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d91f737b-db6a-4b15-ad8c-b8a546d3536c", "id": 3} v 0) v1 -- 0x7f0418069f30 con 0x7f04180a96b0 2024-03-25T17:56:29.159 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:29.159+0000 7fb497bf1040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T17:56:29.167 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:29.166+0000 7fc44ee54040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T17:56:29.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.189+0000 7f03fffff640 1 -- 172.21.15.169:0/3998518633 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d91f737b-db6a-4b15-ad8c-b8a546d3536c", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2911147306 0 525046306) 0x7f040c0064b0 con 0x7f04180a96b0 2024-03-25T17:56:29.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.189+0000 7f03fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d91f737b-db6a-4b15-ad8c-b8a546d3536c", "id": 3}] 2024-03-25T17:56:29.189 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.189+0000 7f03fffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:56:29.189 INFO:teuthology.orchestra.run.smithi169.stdout:3 2024-03-25T17:56:29.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.190+0000 7f041e214640 10 librados: watch_flush enter 2024-03-25T17:56:29.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.190+0000 7f041e214640 10 librados: watch_flush exit 2024-03-25T17:56:29.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.190+0000 7f041e214640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-25T17:56:29.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.190+0000 7f041e214640 10 client.4137.objecter successfully canceled tick 2024-03-25T17:56:29.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.190+0000 7f041e214640 10 monclient: shutdown 2024-03-25T17:56:29.191 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.190+0000 7f041e214640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:29.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.190+0000 7f041e214640 1 -- 172.21.15.169:0/3998518633 >> v1:172.21.15.18:6789/0 conn(0x7f04180a96b0 legacy=0x7f041813b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:29.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.190+0000 7f041e214640 1 -- 172.21.15.169:0/3998518633 shutdown_connections 2024-03-25T17:56:29.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.190+0000 7f041e214640 1 -- 172.21.15.169:0/3998518633 >> 172.21.15.169:0/3998518633 conn(0x7f04180668e0 msgr2=0x7f04180b6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:29.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.191+0000 7f041e214640 1 -- 172.21.15.169:0/3998518633 shutdown_connections 2024-03-25T17:56:29.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.191+0000 7f041e214640 1 -- 172.21.15.169:0/3998518633 wait complete. 2024-03-25T17:56:29.192 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.191+0000 7f041e214640 1 librados: shutdown 2024-03-25T17:56:29.202 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 43877ab5-09f4-45fd-a3bc-bbb38bcb1c04 4 2024-03-25T17:56:29.326 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:29.326+0000 7fb497bf1040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T17:56:29.335 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:29.335+0000 7fc44ee54040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T17:56:29.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.342+0000 7f8d29606640 10 monclient: get_monmap_and_config 2024-03-25T17:56:29.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.342+0000 7f8d29606640 10 monclient: build_initial_monmap 2024-03-25T17:56:29.343 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.342+0000 7f8d29606640 10 monclient: monmap: 2024-03-25T17:56:29.343 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T17:56:29.343 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:29.343 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:56:29.343732+0000 2024-03-25T17:56:29.343 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:56:29.343732+0000 2024-03-25T17:56:29.343 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:29.343 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:29.343 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:29.343 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:56:29.343 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:56:29.343 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:29.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.343+0000 7f8d29606640 1 Processor -- start 2024-03-25T17:56:29.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.343+0000 7f8d29606640 1 -- start start 2024-03-25T17:56:29.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.343+0000 7f8d29606640 10 monclient: init 2024-03-25T17:56:29.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.344+0000 7f8d29606640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:29.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.344+0000 7f8d29606640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:56:29.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.344+0000 7f8d29606640 10 monclient(hunting): picked mon.noname-c con 0x7f8d240b4700 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:29.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.344+0000 7f8d29606640 10 monclient(hunting): picked mon.noname-b con 0x7f8d24142a20 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:29.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.344+0000 7f8d29606640 10 monclient(hunting): picked mon.noname-a con 0x7f8d240a96b0 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:29.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.344+0000 7f8d29606640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d240a9d60 con 0x7f8d240a96b0 2024-03-25T17:56:29.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.344+0000 7f8d29606640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2413ff10 con 0x7f8d240b4700 2024-03-25T17:56:29.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.344+0000 7f8d29606640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d240aa1f0 con 0x7f8d24142a20 2024-03-25T17:56:29.345 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.344+0000 7f8d29606640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:29.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.344+0000 7f8d29606640 10 monclient(hunting): authenticate will time out at 6904.629784s 2024-03-25T17:56:29.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.344+0000 7f8d237fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f8d240a96b0 0x7f8d2413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.169:34704/0 (socket says 172.21.15.169:34704) 2024-03-25T17:56:29.346 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.344+0000 7f8d237fe640 1 -- 172.21.15.169:0/3127306617 learned_addr learned my addr 172.21.15.169:0/3127306617 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T17:56:29.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 1 -- 172.21.15.169:0/3127306617 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255087519 0 0) 0x7f8d2413ff10 con 0x7f8d240b4700 2024-03-25T17:56:29.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:56:29.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:29.347 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:29.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 10 monclient(hunting): my global_id is 4141 2024-03-25T17:56:29.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 1 -- 172.21.15.169:0/3127306617 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d0c003520 con 0x7f8d240b4700 2024-03-25T17:56:29.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 1 -- 172.21.15.169:0/3127306617 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801568746 0 0) 0x7f8d240aa1f0 con 0x7f8d24142a20 2024-03-25T17:56:29.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:56:29.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:29.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:29.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 10 monclient(hunting): my global_id is 4142 2024-03-25T17:56:29.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 1 -- 172.21.15.169:0/3127306617 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d2413ff10 con 0x7f8d24142a20 2024-03-25T17:56:29.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 1 -- 172.21.15.169:0/3127306617 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979486603 0 0) 0x7f8d240a9d60 con 0x7f8d240a96b0 2024-03-25T17:56:29.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:56:29.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:29.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:29.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 10 monclient(hunting): my global_id is 4140 2024-03-25T17:56:29.348 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.347+0000 7f8d21ffb640 1 -- 172.21.15.169:0/3127306617 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d240aa1f0 con 0x7f8d240a96b0 2024-03-25T17:56:29.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.350+0000 7f8d21ffb640 1 -- 172.21.15.169:0/3127306617 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2369222225 0 0) 0x7f8d240aa1f0 con 0x7f8d240a96b0 2024-03-25T17:56:29.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.350+0000 7f8d21ffb640 1 -- 172.21.15.169:0/3127306617 >> v1:172.21.15.18:6790/0 conn(0x7f8d24142a20 legacy=0x7f8d2413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:29.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.350+0000 7f8d21ffb640 1 -- 172.21.15.169:0/3127306617 >> v1:172.21.15.169:6789/0 conn(0x7f8d240b4700 legacy=0x7f8d24142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:29.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.350+0000 7f8d21ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:56:29.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.350+0000 7f8d21ffb640 1 monclient: found mon.noname-a 2024-03-25T17:56:29.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.350+0000 7f8d21ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:29.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.350+0000 7f8d21ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T17:56:29.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.350+0000 7f8d21ffb640 1 -- 172.21.15.169:0/3127306617 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d240b4ae0 con 0x7f8d240a96b0 2024-03-25T17:56:29.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.350+0000 7f8d21ffb640 10 monclient: _finish_auth 0 2024-03-25T17:56:29.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.350+0000 7f8d21ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:56:29.351 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.350+0000 7f8d21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:29.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.350+0000 7f8d21ffb640 1 -- 172.21.15.169:0/3127306617 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8d140015b0 con 0x7f8d240a96b0 2024-03-25T17:56:29.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.350+0000 7f8d21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:29.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.351+0000 7f8d21ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:56:29.352 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.351+0000 7f8d21ffb640 10 monclient: dump: 2024-03-25T17:56:29.352 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:29.352 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:29.352 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.352 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.352 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:29.352 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:29.352 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:29.352 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:29.353 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:29.353 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:29.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.351+0000 7f8d29606640 5 monclient: authenticate success, global_id 4140 2024-03-25T17:56:29.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.351+0000 7f8d29606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:29.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.352+0000 7f8d21ffb640 1 -- 172.21.15.169:0/3127306617 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d14001710 con 0x7f8d240a96b0 2024-03-25T17:56:29.354 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.352+0000 7f8d21ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:29.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.352+0000 7f8d21ffb640 1 -- 172.21.15.169:0/3127306617 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8d14001b60 con 0x7f8d240a96b0 2024-03-25T17:56:29.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.352+0000 7f8d21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:29.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.352+0000 7f8d21ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:29.355 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.352+0000 7f8d21ffb640 10 monclient: dump: 2024-03-25T17:56:29.355 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:29.355 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:29.355 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.355 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.355 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:29.355 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:29.355 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:29.355 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:29.355 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:29.355 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:29.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.352+0000 7f8d29606640 10 monclient: get_monmap_and_config success 2024-03-25T17:56:29.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.352+0000 7f8d29606640 4 set_mon_vals no callback set 2024-03-25T17:56:29.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.353+0000 7f8d29606640 10 monclient: shutdown 2024-03-25T17:56:29.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.353+0000 7f8d29606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:29.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.353+0000 7f8d29606640 1 -- 172.21.15.169:0/3127306617 >> v1:172.21.15.18:6789/0 conn(0x7f8d240a96b0 legacy=0x7f8d2413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:29.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.353+0000 7f8d29606640 1 -- 172.21.15.169:0/3127306617 shutdown_connections 2024-03-25T17:56:29.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.353+0000 7f8d29606640 1 -- 172.21.15.169:0/3127306617 >> 172.21.15.169:0/3127306617 conn(0x7f8d240668e0 msgr2=0x7f8d24066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:29.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.353+0000 7f8d29606640 1 -- 172.21.15.169:0/3127306617 shutdown_connections 2024-03-25T17:56:29.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.353+0000 7f8d29606640 1 -- 172.21.15.169:0/3127306617 wait complete. 2024-03-25T17:56:29.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.354+0000 7f8d29606640 10 monclient: build_initial_monmap 2024-03-25T17:56:29.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.354+0000 7f8d29606640 10 monclient: monmap: 2024-03-25T17:56:29.356 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T17:56:29.356 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:56:29.356 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-25T17:56:29.357 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-25T17:56:29.357 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:29.357 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-25T17:56:29.357 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:29.357 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:56:29.357 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:56:29.357 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:29.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.355+0000 7f8d29606640 1 librados: starting msgr at 2024-03-25T17:56:29.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.355+0000 7f8d29606640 1 librados: starting objecter 2024-03-25T17:56:29.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.355+0000 7f8d29606640 1 Processor -- start 2024-03-25T17:56:29.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.355+0000 7f8d29606640 1 -- start start 2024-03-25T17:56:29.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.355+0000 7f8d29606640 1 librados: setting wanted keys 2024-03-25T17:56:29.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.355+0000 7f8d29606640 1 librados: calling monclient init 2024-03-25T17:56:29.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.355+0000 7f8d29606640 10 monclient: init 2024-03-25T17:56:29.359 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.356+0000 7f8d29606640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:29.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.356+0000 7f8d29606640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:56:29.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.356+0000 7f8d29606640 10 monclient(hunting): picked mon.noname-c con 0x7f8d240a96b0 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:29.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.356+0000 7f8d29606640 10 monclient(hunting): picked mon.noname-a con 0x7f8d240b4700 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:29.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.356+0000 7f8d29606640 10 monclient(hunting): picked mon.noname-b con 0x7f8d24142a20 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:29.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.356+0000 7f8d29606640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d24131080 con 0x7f8d240b4700 2024-03-25T17:56:29.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.356+0000 7f8d29606640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d24133d30 con 0x7f8d24142a20 2024-03-25T17:56:29.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.356+0000 7f8d29606640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d24134010 con 0x7f8d240a96b0 2024-03-25T17:56:29.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.356+0000 7f8d29606640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:29.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.356+0000 7f8d29606640 10 monclient(hunting): authenticate will time out at 6904.642047s 2024-03-25T17:56:29.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.357+0000 7f8d227fc640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f8d240b4700 0x7f8d241349f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.169:34712/0 (socket says 172.21.15.169:34712) 2024-03-25T17:56:29.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.357+0000 7f8d227fc640 1 -- 172.21.15.169:0/1447679736 learned_addr learned my addr 172.21.15.169:0/1447679736 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T17:56:29.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.357+0000 7f8d0b7fe640 10 client.?.objecter ms_handle_connect 0x7f8d240b4700 2024-03-25T17:56:29.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.357+0000 7f8d0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:29.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.357+0000 7f8d0b7fe640 10 client.?.objecter ms_handle_connect 0x7f8d24142a20 2024-03-25T17:56:29.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.357+0000 7f8d0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:29.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.357+0000 7f8d0b7fe640 10 client.?.objecter ms_handle_connect 0x7f8d240a96b0 2024-03-25T17:56:29.361 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.357+0000 7f8d0b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:29.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.359+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964661923 0 0) 0x7f8d24133d30 con 0x7f8d24142a20 2024-03-25T17:56:29.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.359+0000 7f8d0b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:56:29.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.359+0000 7f8d0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:29.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.359+0000 7f8d0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:29.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.359+0000 7f8d0b7fe640 10 monclient(hunting): my global_id is 4144 2024-03-25T17:56:29.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.359+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d10004eb0 con 0x7f8d24142a20 2024-03-25T17:56:29.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.360+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033435161 0 0) 0x7f8d24134010 con 0x7f8d240a96b0 2024-03-25T17:56:29.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.360+0000 7f8d0b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:56:29.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.360+0000 7f8d0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:29.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.360+0000 7f8d0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:29.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.360+0000 7f8d0b7fe640 10 monclient(hunting): my global_id is 4145 2024-03-25T17:56:29.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.360+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d24133d30 con 0x7f8d240a96b0 2024-03-25T17:56:29.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.360+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603638102 0 0) 0x7f8d24131080 con 0x7f8d240b4700 2024-03-25T17:56:29.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.360+0000 7f8d0b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:56:29.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.360+0000 7f8d0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:29.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.360+0000 7f8d0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:29.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.360+0000 7f8d0b7fe640 10 monclient(hunting): my global_id is 4143 2024-03-25T17:56:29.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.360+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d24134010 con 0x7f8d240b4700 2024-03-25T17:56:29.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.365+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 886760479 0 0) 0x7f8d24133d30 con 0x7f8d240a96b0 2024-03-25T17:56:29.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.365+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d24131080 con 0x7f8d240a96b0 2024-03-25T17:56:29.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.365+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8d1c001950 con 0x7f8d240a96b0 2024-03-25T17:56:29.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.365+0000 7f8d0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:29.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.365+0000 7f8d0b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:56:29.366 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.365+0000 7f8d0b7fe640 10 monclient(hunting): dump: 2024-03-25T17:56:29.366 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:29.366 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:29.366 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.367 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.367 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:29.367 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:29.367 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:29.367 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:29.367 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:29.367 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:29.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.365+0000 7f8d0b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:56:29.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.365+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1166797034 0 0) 0x7f8d24134010 con 0x7f8d240b4700 2024-03-25T17:56:29.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.365+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d24133d30 con 0x7f8d240b4700 2024-03-25T17:56:29.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.365+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8d18001cb0 con 0x7f8d240b4700 2024-03-25T17:56:29.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.365+0000 7f8d0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:29.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.365+0000 7f8d0b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:29.367 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.365+0000 7f8d0b7fe640 10 monclient(hunting): dump: 2024-03-25T17:56:29.367 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:29.368 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:29.368 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.368 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.368 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:29.368 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:29.368 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:29.368 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:29.368 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:29.368 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:29.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.366+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1971906252 0 0) 0x7f8d10004eb0 con 0x7f8d24142a20 2024-03-25T17:56:29.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.366+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d24134010 con 0x7f8d24142a20 2024-03-25T17:56:29.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.366+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8d14001a60 con 0x7f8d24142a20 2024-03-25T17:56:29.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.366+0000 7f8d0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:29.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.366+0000 7f8d0b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:56:29.369 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.366+0000 7f8d0b7fe640 10 monclient(hunting): dump: 2024-03-25T17:56:29.369 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:29.369 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:29.369 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.369 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.369 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:29.369 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:29.369 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:29.369 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:29.370 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:29.370 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:29.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2892117981 0 0) 0x7f8d24133d30 con 0x7f8d240b4700 2024-03-25T17:56:29.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 >> v1:172.21.15.18:6790/0 conn(0x7f8d240a96b0 legacy=0x7f8d2412c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:29.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 >> v1:172.21.15.169:6789/0 conn(0x7f8d24142a20 legacy=0x7f8d24135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:29.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d0b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:56:29.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d0b7fe640 1 monclient: found mon.a 2024-03-25T17:56:29.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:29.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d0b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:29.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d241342f0 con 0x7f8d240b4700 2024-03-25T17:56:29.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d0b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:56:29.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d0b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:56:29.371 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:29.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d29606640 5 monclient: authenticate success, global_id 4143 2024-03-25T17:56:29.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d29606640 10 monclient: _renew_subs 2024-03-25T17:56:29.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d29606640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:29.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d29606640 1 -- 172.21.15.169:0/1447679736 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d241345d0 con 0x7f8d240b4700 2024-03-25T17:56:29.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d29606640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:56:29.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d29606640 10 monclient: _renew_subs 2024-03-25T17:56:29.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d29606640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:29.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d29606640 1 -- 172.21.15.169:0/1447679736 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d24135810 con 0x7f8d240b4700 2024-03-25T17:56:29.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.369+0000 7f8d29606640 1 librados: init done 2024-03-25T17:56:29.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.370+0000 7f8d29606640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:29.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.370+0000 7f8d29606640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:29.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.370+0000 7f8d29606640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:29.372 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.370+0000 7f8d29606640 1 -- 172.21.15.169:0/1447679736 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2404f730 con 0x7f8d240b4700 2024-03-25T17:56:29.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.373+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d18003990 con 0x7f8d240b4700 2024-03-25T17:56:29.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.373+0000 7f8d0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:29.373 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.373+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8d18003de0 con 0x7f8d240b4700 2024-03-25T17:56:29.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.373+0000 7f8d0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:29.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.373+0000 7f8d0b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:29.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.373+0000 7f8d0b7fe640 10 monclient: dump: 2024-03-25T17:56:29.374 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:29.374 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:29.374 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.374 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.374 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:29.374 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:29.374 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:29.374 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:29.374 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:29.374 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:29.374 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.373+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8d18004390 con 0x7f8d240b4700 2024-03-25T17:56:29.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.373+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3504149928 0 0) 0x7f8d18006970 con 0x7f8d240b4700 2024-03-25T17:56:29.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.373+0000 7f8d0b7fe640 10 client.4143.objecter ms_dispatch 0x7f8d24000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T17:56:29.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.373+0000 7f8d0b7fe640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T17:56:29.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.373+0000 7f8d0b7fe640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-03-25T17:56:29.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.373+0000 7f8d217fa640 4 set_mon_vals no callback set 2024-03-25T17:56:29.375 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.373+0000 7f8d0b7fe640 20 client.4143.objecter dump_active .. 0 homeless 2024-03-25T17:56:29.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.475+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8d18004640 con 0x7f8d240b4700 2024-03-25T17:56:29.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.475+0000 7f8d0b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:29.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.475+0000 7f8d0b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:56:29.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.549+0000 7f8d29606640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "43877ab5-09f4-45fd-a3bc-bbb38bcb1c04", "id": 4}] 2024-03-25T17:56:29.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.549+0000 7f8d29606640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "43877ab5-09f4-45fd-a3bc-bbb38bcb1c04", "id": 4}] 2024-03-25T17:56:29.549 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.549+0000 7f8d29606640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:29.550 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.549+0000 7f8d29606640 1 -- 172.21.15.169:0/1447679736 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd new", "uuid": "43877ab5-09f4-45fd-a3bc-bbb38bcb1c04", "id": 4} v 0) v1 -- 0x7f8d240b5b60 con 0x7f8d240b4700 2024-03-25T17:56:29.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.597+0000 7f8d0b7fe640 1 -- 172.21.15.169:0/1447679736 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "43877ab5-09f4-45fd-a3bc-bbb38bcb1c04", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 465213040 0 1697012583) 0x7f8d180047e0 con 0x7f8d240b4700 2024-03-25T17:56:29.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.597+0000 7f8d0b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "43877ab5-09f4-45fd-a3bc-bbb38bcb1c04", "id": 4}] 2024-03-25T17:56:29.597 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.597+0000 7f8d0b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:56:29.597 INFO:teuthology.orchestra.run.smithi169.stdout:4 2024-03-25T17:56:29.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.598+0000 7f8d29606640 10 librados: watch_flush enter 2024-03-25T17:56:29.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.598+0000 7f8d29606640 10 librados: watch_flush exit 2024-03-25T17:56:29.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.598+0000 7f8d29606640 20 client.4143.objecter shutdown clearing up homeless session... 2024-03-25T17:56:29.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.598+0000 7f8d29606640 10 client.4143.objecter successfully canceled tick 2024-03-25T17:56:29.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.598+0000 7f8d29606640 10 monclient: shutdown 2024-03-25T17:56:29.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.598+0000 7f8d29606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:29.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.598+0000 7f8d29606640 1 -- 172.21.15.169:0/1447679736 >> v1:172.21.15.18:6789/0 conn(0x7f8d240b4700 legacy=0x7f8d241349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:29.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.598+0000 7f8d29606640 1 -- 172.21.15.169:0/1447679736 shutdown_connections 2024-03-25T17:56:29.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.598+0000 7f8d29606640 1 -- 172.21.15.169:0/1447679736 >> 172.21.15.169:0/1447679736 conn(0x7f8d240668e0 msgr2=0x7f8d241364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:29.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.598+0000 7f8d29606640 1 -- 172.21.15.169:0/1447679736 shutdown_connections 2024-03-25T17:56:29.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.598+0000 7f8d29606640 1 -- 172.21.15.169:0/1447679736 wait complete. 2024-03-25T17:56:29.600 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.598+0000 7f8d29606640 1 librados: shutdown 2024-03-25T17:56:29.608 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb 5 2024-03-25T17:56:29.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.708+0000 7f10d17f6640 10 monclient: get_monmap_and_config 2024-03-25T17:56:29.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.708+0000 7f10d17f6640 10 monclient: build_initial_monmap 2024-03-25T17:56:29.708 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.708+0000 7f10d17f6640 10 monclient: monmap: 2024-03-25T17:56:29.709 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T17:56:29.709 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:29.709 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:56:29.709382+0000 2024-03-25T17:56:29.709 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:56:29.709382+0000 2024-03-25T17:56:29.709 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:29.709 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:29.709 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:29.709 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:56:29.709 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:56:29.709 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:29.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.708+0000 7f10d17f6640 1 Processor -- start 2024-03-25T17:56:29.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.709+0000 7f10d17f6640 1 -- start start 2024-03-25T17:56:29.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.709+0000 7f10d17f6640 10 monclient: init 2024-03-25T17:56:29.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.709+0000 7f10d17f6640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:29.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.709+0000 7f10d17f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:56:29.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.709+0000 7f10d17f6640 10 monclient(hunting): picked mon.noname-c con 0x7f10cc0a9920 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:29.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.709+0000 7f10d17f6640 10 monclient(hunting): picked mon.noname-b con 0x7f10cc0b45a0 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:29.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.709+0000 7f10d17f6640 10 monclient(hunting): picked mon.noname-a con 0x7f10cc146450 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:29.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.709+0000 7f10d17f6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc06b100 con 0x7f10cc146450 2024-03-25T17:56:29.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.709+0000 7f10d17f6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc0b2940 con 0x7f10cc0a9920 2024-03-25T17:56:29.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.709+0000 7f10d17f6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc13fec0 con 0x7f10cc0b45a0 2024-03-25T17:56:29.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.709+0000 7f10d17f6640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:29.711 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.709+0000 7f10d17f6640 10 monclient(hunting): authenticate will time out at 6904.995012s 2024-03-25T17:56:29.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.710+0000 7f10cb7fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f10cc146450 0x7f10cc14dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.169:34728/0 (socket says 172.21.15.169:34728) 2024-03-25T17:56:29.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.710+0000 7f10cb7fe640 1 -- 172.21.15.169:0/939378565 learned_addr learned my addr 172.21.15.169:0/939378565 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T17:56:29.712 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.712+0000 7f10c9ffb640 1 -- 172.21.15.169:0/939378565 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950797342 0 0) 0x7f10cc06b100 con 0x7f10cc146450 2024-03-25T17:56:29.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.712+0000 7f10c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:56:29.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.712+0000 7f10c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:29.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.712+0000 7f10c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:29.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.712+0000 7f10c9ffb640 10 monclient(hunting): my global_id is 4146 2024-03-25T17:56:29.713 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.712+0000 7f10c9ffb640 1 -- 172.21.15.169:0/939378565 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b4003520 con 0x7f10cc146450 2024-03-25T17:56:29.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.713+0000 7f10c9ffb640 1 -- 172.21.15.169:0/939378565 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007306884 0 0) 0x7f10cc0b2940 con 0x7f10cc0a9920 2024-03-25T17:56:29.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.713+0000 7f10c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:56:29.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.713+0000 7f10c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:29.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.713+0000 7f10c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:29.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.713+0000 7f10c9ffb640 10 monclient(hunting): my global_id is 4147 2024-03-25T17:56:29.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.713+0000 7f10c9ffb640 1 -- 172.21.15.169:0/939378565 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10cc06b100 con 0x7f10cc0a9920 2024-03-25T17:56:29.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.713+0000 7f10c9ffb640 1 -- 172.21.15.169:0/939378565 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029849777 0 0) 0x7f10cc13fec0 con 0x7f10cc0b45a0 2024-03-25T17:56:29.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.713+0000 7f10c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:56:29.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.713+0000 7f10c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:29.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.713+0000 7f10c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:29.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.713+0000 7f10c9ffb640 10 monclient(hunting): my global_id is 4148 2024-03-25T17:56:29.714 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.713+0000 7f10c9ffb640 1 -- 172.21.15.169:0/939378565 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10cc0b2940 con 0x7f10cc0b45a0 2024-03-25T17:56:29.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10c9ffb640 1 -- 172.21.15.169:0/939378565 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2155979485 0 0) 0x7f10b4003520 con 0x7f10cc146450 2024-03-25T17:56:29.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10c9ffb640 1 -- 172.21.15.169:0/939378565 >> v1:172.21.15.18:6790/0 conn(0x7f10cc0b45a0 legacy=0x7f10cc145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:29.716 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10c9ffb640 1 -- 172.21.15.169:0/939378565 >> v1:172.21.15.169:6789/0 conn(0x7f10cc0a9920 legacy=0x7f10cc13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:29.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10c9ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:56:29.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10c9ffb640 1 monclient: found mon.noname-a 2024-03-25T17:56:29.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:29.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10c9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T17:56:29.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10c9ffb640 1 -- 172.21.15.169:0/939378565 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc140050 con 0x7f10cc146450 2024-03-25T17:56:29.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10c9ffb640 10 monclient: _finish_auth 0 2024-03-25T17:56:29.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10c9ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:56:29.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:29.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10d17f6640 5 monclient: authenticate success, global_id 4146 2024-03-25T17:56:29.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10d17f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:29.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10c9ffb640 1 -- 172.21.15.169:0/939378565 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f10c4001830 con 0x7f10cc146450 2024-03-25T17:56:29.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:29.717 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10c9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:56:29.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10c9ffb640 10 monclient: dump: 2024-03-25T17:56:29.718 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:29.718 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:29.718 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.718 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.718 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:29.718 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:29.718 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:29.718 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:29.718 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:29.718 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:29.718 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.716+0000 7f10d17f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:29.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.717+0000 7f10c9ffb640 1 -- 172.21.15.169:0/939378565 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10c4001a80 con 0x7f10cc146450 2024-03-25T17:56:29.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.717+0000 7f10c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:29.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.717+0000 7f10d17f6640 10 monclient: get_monmap_and_config success 2024-03-25T17:56:29.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.717+0000 7f10d17f6640 4 set_mon_vals no callback set 2024-03-25T17:56:29.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.718+0000 7f10c9ffb640 1 -- 172.21.15.169:0/939378565 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f10c4001e10 con 0x7f10cc146450 2024-03-25T17:56:29.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.718+0000 7f10c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:29.721 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.718+0000 7f10c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:29.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.718+0000 7f10c9ffb640 10 monclient: dump: 2024-03-25T17:56:29.722 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:29.722 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:29.722 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.722 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.722 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:29.722 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:29.722 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:29.722 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:29.722 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:29.722 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:29.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.718+0000 7f10d17f6640 10 monclient: shutdown 2024-03-25T17:56:29.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.718+0000 7f10d17f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:29.722 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.718+0000 7f10d17f6640 1 -- 172.21.15.169:0/939378565 >> v1:172.21.15.18:6789/0 conn(0x7f10cc146450 legacy=0x7f10cc14dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:29.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.718+0000 7f10d17f6640 1 -- 172.21.15.169:0/939378565 shutdown_connections 2024-03-25T17:56:29.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.718+0000 7f10d17f6640 1 -- 172.21.15.169:0/939378565 >> 172.21.15.169:0/939378565 conn(0x7f10cc0b7f40 msgr2=0x7f10cc0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:29.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.718+0000 7f10d17f6640 1 -- 172.21.15.169:0/939378565 shutdown_connections 2024-03-25T17:56:29.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.718+0000 7f10d17f6640 1 -- 172.21.15.169:0/939378565 wait complete. 2024-03-25T17:56:29.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.719+0000 7f10d17f6640 10 monclient: build_initial_monmap 2024-03-25T17:56:29.723 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.719+0000 7f10d17f6640 10 monclient: monmap: 2024-03-25T17:56:29.723 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T17:56:29.723 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:56:29.723 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-25T17:56:29.723 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-25T17:56:29.723 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:29.723 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-25T17:56:29.723 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:29.723 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:56:29.724 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:56:29.724 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:29.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.719+0000 7f10d17f6640 1 librados: starting msgr at 2024-03-25T17:56:29.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.719+0000 7f10d17f6640 1 librados: starting objecter 2024-03-25T17:56:29.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.719+0000 7f10d17f6640 1 Processor -- start 2024-03-25T17:56:29.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.720+0000 7f10d17f6640 1 -- start start 2024-03-25T17:56:29.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.720+0000 7f10d17f6640 1 librados: setting wanted keys 2024-03-25T17:56:29.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.720+0000 7f10d17f6640 1 librados: calling monclient init 2024-03-25T17:56:29.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.720+0000 7f10d17f6640 10 monclient: init 2024-03-25T17:56:29.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.720+0000 7f10d17f6640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:29.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.720+0000 7f10d17f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:56:29.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.720+0000 7f10d17f6640 10 monclient(hunting): picked mon.noname-b con 0x7f10cc0a9920 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:29.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.720+0000 7f10d17f6640 10 monclient(hunting): picked mon.noname-a con 0x7f10cc0b45a0 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:29.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.720+0000 7f10d17f6640 10 monclient(hunting): picked mon.noname-c con 0x7f10cc146450 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:29.724 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.720+0000 7f10d17f6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc1401f0 con 0x7f10cc0b45a0 2024-03-25T17:56:29.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.720+0000 7f10d17f6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc1d80e0 con 0x7f10cc0a9920 2024-03-25T17:56:29.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.720+0000 7f10d17f6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc1d8270 con 0x7f10cc146450 2024-03-25T17:56:29.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.720+0000 7f10d17f6640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:29.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.720+0000 7f10d17f6640 10 monclient(hunting): authenticate will time out at 6905.005976s 2024-03-25T17:56:29.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.720+0000 7f10ca7fc640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f10cc0b45a0 0x7f10cc137170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.169:34740/0 (socket says 172.21.15.169:34740) 2024-03-25T17:56:29.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.720+0000 7f10ca7fc640 1 -- 172.21.15.169:0/3680528725 learned_addr learned my addr 172.21.15.169:0/3680528725 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T17:56:29.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.721+0000 7f10b37fe640 10 client.?.objecter ms_handle_connect 0x7f10cc0b45a0 2024-03-25T17:56:29.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.721+0000 7f10b37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:29.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.721+0000 7f10b37fe640 10 client.?.objecter ms_handle_connect 0x7f10cc0a9920 2024-03-25T17:56:29.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.721+0000 7f10b37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:29.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.721+0000 7f10b37fe640 10 client.?.objecter ms_handle_connect 0x7f10cc146450 2024-03-25T17:56:29.725 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.721+0000 7f10b37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:29.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093867833 0 0) 0x7f10cc1d80e0 con 0x7f10cc0a9920 2024-03-25T17:56:29.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:56:29.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:29.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:29.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 10 monclient(hunting): my global_id is 4150 2024-03-25T17:56:29.726 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b8004eb0 con 0x7f10cc0a9920 2024-03-25T17:56:29.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456044863 0 0) 0x7f10cc1401f0 con 0x7f10cc0b45a0 2024-03-25T17:56:29.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:56:29.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:29.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:29.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 10 monclient(hunting): my global_id is 4149 2024-03-25T17:56:29.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10cc1d80e0 con 0x7f10cc0b45a0 2024-03-25T17:56:29.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748570459 0 0) 0x7f10cc1d8270 con 0x7f10cc146450 2024-03-25T17:56:29.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:56:29.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:29.727 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:29.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 10 monclient(hunting): my global_id is 4151 2024-03-25T17:56:29.728 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.723+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10cc1401f0 con 0x7f10cc146450 2024-03-25T17:56:29.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.727+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 572782372 0 0) 0x7f10b8004eb0 con 0x7f10cc0a9920 2024-03-25T17:56:29.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.727+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10cc1d8270 con 0x7f10cc0a9920 2024-03-25T17:56:29.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.727+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f10bc0018b0 con 0x7f10cc0a9920 2024-03-25T17:56:29.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.727+0000 7f10b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:29.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.727+0000 7f10b37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:56:29.729 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.727+0000 7f10b37fe640 10 monclient(hunting): dump: 2024-03-25T17:56:29.730 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:29.730 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:29.730 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.730 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.730 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:29.730 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:29.730 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:29.730 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:29.730 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:29.730 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:29.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.727+0000 7f10b37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:56:29.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.728+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2547488548 0 0) 0x7f10cc1d80e0 con 0x7f10cc0b45a0 2024-03-25T17:56:29.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.728+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10b8004eb0 con 0x7f10cc0b45a0 2024-03-25T17:56:29.730 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.728+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f10c00019f0 con 0x7f10cc0b45a0 2024-03-25T17:56:29.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.728+0000 7f10b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:29.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.728+0000 7f10b37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:29.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.728+0000 7f10b37fe640 10 monclient(hunting): dump: 2024-03-25T17:56:29.731 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:29.731 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:29.731 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.731 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.731 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:29.731 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:29.731 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:29.731 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:29.731 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:29.731 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:29.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.729+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2080417090 0 0) 0x7f10cc1401f0 con 0x7f10cc146450 2024-03-25T17:56:29.731 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.729+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10cc1d80e0 con 0x7f10cc146450 2024-03-25T17:56:29.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.729+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f10c4001830 con 0x7f10cc146450 2024-03-25T17:56:29.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.729+0000 7f10b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:29.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.729+0000 7f10b37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:56:29.732 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.729+0000 7f10b37fe640 10 monclient(hunting): dump: 2024-03-25T17:56:29.732 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:29.732 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:29.732 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.732 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.732 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:29.732 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:29.732 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:29.733 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:29.733 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:29.733 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:29.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1068572018 0 0) 0x7f10cc1d8270 con 0x7f10cc0a9920 2024-03-25T17:56:29.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 >> v1:172.21.15.18:6790/0 conn(0x7f10cc146450 legacy=0x7f10cc140680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:29.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 >> v1:172.21.15.18:6789/0 conn(0x7f10cc0b45a0 legacy=0x7f10cc137170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:29.734 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10b37fe640 10 monclient: _finish_hunting 0 2024-03-25T17:56:29.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10b37fe640 1 monclient: found mon.b 2024-03-25T17:56:29.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:29.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10b37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:29.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc137880 con 0x7f10cc0a9920 2024-03-25T17:56:29.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10b37fe640 10 monclient: _finish_auth 0 2024-03-25T17:56:29.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10b37fe640 10 monclient: _check_auth_tickets 2024-03-25T17:56:29.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:29.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10d17f6640 5 monclient: authenticate success, global_id 4150 2024-03-25T17:56:29.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10d17f6640 10 monclient: _renew_subs 2024-03-25T17:56:29.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10d17f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:29.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10d17f6640 1 -- 172.21.15.169:0/3680528725 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10cc137a70 con 0x7f10cc0a9920 2024-03-25T17:56:29.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10d17f6640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:56:29.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10d17f6640 10 monclient: _renew_subs 2024-03-25T17:56:29.735 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10d17f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:29.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10d17f6640 1 -- 172.21.15.169:0/3680528725 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10cc13c1c0 con 0x7f10cc0a9920 2024-03-25T17:56:29.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.731+0000 7f10d17f6640 1 librados: init done 2024-03-25T17:56:29.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.732+0000 7f10d17f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:29.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.732+0000 7f10d17f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:29.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.732+0000 7f10d17f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:29.736 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.732+0000 7f10d17f6640 1 -- 172.21.15.169:0/3680528725 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10cc1403a0 con 0x7f10cc0a9920 2024-03-25T17:56:29.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.832+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10bc0035f0 con 0x7f10cc0a9920 2024-03-25T17:56:29.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.832+0000 7f10b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:29.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.832+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f10bc003760 con 0x7f10cc0a9920 2024-03-25T17:56:29.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.832+0000 7f10b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:29.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.832+0000 7f10b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:56:29.833 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.832+0000 7f10b37fe640 10 monclient: dump: 2024-03-25T17:56:29.833 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:29.833 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:29.833 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.833 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:29.834 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:29.834 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:29.834 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:29.834 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:29.834 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:29.834 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:29.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.832+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f10bc003c50 con 0x7f10cc0a9920 2024-03-25T17:56:29.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.832+0000 7f10c97fa640 4 set_mon_vals no callback set 2024-03-25T17:56:29.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.832+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2898705251 0 0) 0x7f10bc005310 con 0x7f10cc0a9920 2024-03-25T17:56:29.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.832+0000 7f10b37fe640 10 client.4150.objecter ms_dispatch 0x7f10cc000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T17:56:29.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.832+0000 7f10b37fe640 3 client.4150.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T17:56:29.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.832+0000 7f10b37fe640 3 client.4150.objecter handle_osd_map decoding full epoch 7 2024-03-25T17:56:29.834 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.832+0000 7f10b37fe640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-25T17:56:29.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.832+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f10bc005650 con 0x7f10cc0a9920 2024-03-25T17:56:29.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.832+0000 7f10b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:29.835 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.832+0000 7f10b37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:56:29.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.904+0000 7f10d17f6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb", "id": 5}] 2024-03-25T17:56:29.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.904+0000 7f10d17f6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb", "id": 5}] 2024-03-25T17:56:29.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.904+0000 7f10d17f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:29.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.904+0000 7f10d17f6640 1 -- 172.21.15.169:0/3680528725 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb", "id": 5} v 0) v1 -- 0x7f10cc0ac2b0 con 0x7f10cc0a9920 2024-03-25T17:56:29.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.970+0000 7f10b37fe640 1 -- 172.21.15.169:0/3680528725 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3738608543 0 1988412176) 0x7f10bc004e20 con 0x7f10cc0a9920 2024-03-25T17:56:29.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.970+0000 7f10b37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb", "id": 5}] 2024-03-25T17:56:29.970 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.970+0000 7f10b37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:56:29.971 INFO:teuthology.orchestra.run.smithi169.stdout:5 2024-03-25T17:56:29.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.971+0000 7f10d17f6640 10 librados: watch_flush enter 2024-03-25T17:56:29.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.971+0000 7f10d17f6640 10 librados: watch_flush exit 2024-03-25T17:56:29.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.971+0000 7f10d17f6640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-25T17:56:29.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.971+0000 7f10d17f6640 10 client.4150.objecter successfully canceled tick 2024-03-25T17:56:29.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.971+0000 7f10d17f6640 10 monclient: shutdown 2024-03-25T17:56:29.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.971+0000 7f10d17f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:29.972 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.971+0000 7f10d17f6640 1 -- 172.21.15.169:0/3680528725 >> v1:172.21.15.169:6789/0 conn(0x7f10cc0a9920 legacy=0x7f10cc1d79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:29.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.971+0000 7f10d17f6640 1 -- 172.21.15.169:0/3680528725 shutdown_connections 2024-03-25T17:56:29.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.971+0000 7f10d17f6640 1 -- 172.21.15.169:0/3680528725 >> 172.21.15.169:0/3680528725 conn(0x7f10cc0b7f40 msgr2=0x7f10cc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:29.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.971+0000 7f10d17f6640 1 -- 172.21.15.169:0/3680528725 shutdown_connections 2024-03-25T17:56:29.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.971+0000 7f10d17f6640 1 -- 172.21.15.169:0/3680528725 wait complete. 2024-03-25T17:56:29.973 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:29.971+0000 7f10d17f6640 1 librados: shutdown 2024-03-25T17:56:29.982 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new c3bd1d77-7390-4452-8980-b0ef85bc74a8 6 2024-03-25T17:56:30.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.083+0000 7f60c88b3640 10 monclient: get_monmap_and_config 2024-03-25T17:56:30.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.083+0000 7f60c88b3640 10 monclient: build_initial_monmap 2024-03-25T17:56:30.084 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.083+0000 7f60c88b3640 10 monclient: monmap: 2024-03-25T17:56:30.084 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T17:56:30.084 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:30.084 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:56:30.084579+0000 2024-03-25T17:56:30.084 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:56:30.084579+0000 2024-03-25T17:56:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:56:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:56:30.085 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.084+0000 7f60c88b3640 1 Processor -- start 2024-03-25T17:56:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.084+0000 7f60c88b3640 1 -- start start 2024-03-25T17:56:30.085 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.084+0000 7f60c88b3640 10 monclient: init 2024-03-25T17:56:30.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.084+0000 7f60c88b3640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:30.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.084+0000 7f60c88b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:56:30.086 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.084+0000 7f60c88b3640 10 monclient(hunting): picked mon.noname-c con 0x7f60c40b4700 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:30.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.084+0000 7f60c88b3640 10 monclient(hunting): picked mon.noname-a con 0x7f60c4142a20 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:30.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.084+0000 7f60c88b3640 10 monclient(hunting): picked mon.noname-b con 0x7f60c40a96b0 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:30.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.084+0000 7f60c88b3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c40a9d60 con 0x7f60c4142a20 2024-03-25T17:56:30.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.084+0000 7f60c88b3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c413ff10 con 0x7f60c40b4700 2024-03-25T17:56:30.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.084+0000 7f60c88b3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c40aa1f0 con 0x7f60c40a96b0 2024-03-25T17:56:30.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.084+0000 7f60c88b3640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:30.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.084+0000 7f60c88b3640 10 monclient(hunting): authenticate will time out at 6905.370330s 2024-03-25T17:56:30.087 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.085+0000 7f60bbfff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f60c4142a20 0x7f60c413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.169:34754/0 (socket says 172.21.15.169:34754) 2024-03-25T17:56:30.088 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.085+0000 7f60bbfff640 1 -- 172.21.15.169:0/2434008311 learned_addr learned my addr 172.21.15.169:0/2434008311 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T17:56:30.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 1 -- 172.21.15.169:0/2434008311 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053053319 0 0) 0x7f60c40a9d60 con 0x7f60c4142a20 2024-03-25T17:56:30.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:56:30.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:30.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:30.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 10 monclient(hunting): my global_id is 4152 2024-03-25T17:56:30.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 1 -- 172.21.15.169:0/2434008311 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60b0003520 con 0x7f60c4142a20 2024-03-25T17:56:30.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 1 -- 172.21.15.169:0/2434008311 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210206560 0 0) 0x7f60c413ff10 con 0x7f60c40b4700 2024-03-25T17:56:30.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:56:30.089 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 10 monclient(hunting): my global_id is 4153 2024-03-25T17:56:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 1 -- 172.21.15.169:0/2434008311 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c40a9d60 con 0x7f60c40b4700 2024-03-25T17:56:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 1 -- 172.21.15.169:0/2434008311 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390511474 0 0) 0x7f60c40aa1f0 con 0x7f60c40a96b0 2024-03-25T17:56:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:56:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 10 monclient(hunting): my global_id is 4154 2024-03-25T17:56:30.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.088+0000 7f60c1d9b640 1 -- 172.21.15.169:0/2434008311 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c413ff10 con 0x7f60c40a96b0 2024-03-25T17:56:30.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.091+0000 7f60c1d9b640 1 -- 172.21.15.169:0/2434008311 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4215979410 0 0) 0x7f60c40a9d60 con 0x7f60c40b4700 2024-03-25T17:56:30.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.091+0000 7f60c1d9b640 1 -- 172.21.15.169:0/2434008311 >> v1:172.21.15.18:6790/0 conn(0x7f60c40a96b0 legacy=0x7f60c413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:30.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.091+0000 7f60c1d9b640 1 -- 172.21.15.169:0/2434008311 >> v1:172.21.15.18:6789/0 conn(0x7f60c4142a20 legacy=0x7f60c413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.091+0000 7f60c1d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:56:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.091+0000 7f60c1d9b640 1 monclient: found mon.noname-c 2024-03-25T17:56:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.091+0000 7f60c1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.091+0000 7f60c1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T17:56:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.091+0000 7f60c1d9b640 1 -- 172.21.15.169:0/2434008311 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c40b4ae0 con 0x7f60c40b4700 2024-03-25T17:56:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.092+0000 7f60c1d9b640 10 monclient: _finish_auth 0 2024-03-25T17:56:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.092+0000 7f60c1d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:56:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.092+0000 7f60c1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.092+0000 7f60c1d9b640 1 -- 172.21.15.169:0/2434008311 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f60b4001670 con 0x7f60c40b4700 2024-03-25T17:56:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.092+0000 7f60c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.092+0000 7f60c1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:56:30.093 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.092+0000 7f60c1d9b640 10 monclient: dump: 2024-03-25T17:56:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:30.094 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:30.094 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.092+0000 7f60c88b3640 5 monclient: authenticate success, global_id 4153 2024-03-25T17:56:30.095 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.092+0000 7f60c88b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:30.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.093+0000 7f60c1d9b640 1 -- 172.21.15.169:0/2434008311 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60b40017d0 con 0x7f60c40b4700 2024-03-25T17:56:30.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.093+0000 7f60c1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:30.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.093+0000 7f60c1d9b640 1 -- 172.21.15.169:0/2434008311 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f60b4001c20 con 0x7f60c40b4700 2024-03-25T17:56:30.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.093+0000 7f60c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:30.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.093+0000 7f60c1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:56:30.098 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.093+0000 7f60c1d9b640 10 monclient: dump: 2024-03-25T17:56:30.098 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:30.098 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:30.098 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.098 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.098 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:30.098 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:30.098 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:30.098 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:30.098 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:30.099 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:30.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.093+0000 7f60c88b3640 10 monclient: get_monmap_and_config success 2024-03-25T17:56:30.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.093+0000 7f60c88b3640 4 set_mon_vals no callback set 2024-03-25T17:56:30.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.094+0000 7f60c88b3640 10 monclient: shutdown 2024-03-25T17:56:30.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.094+0000 7f60c88b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:30.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.094+0000 7f60c88b3640 1 -- 172.21.15.169:0/2434008311 >> v1:172.21.15.169:6789/0 conn(0x7f60c40b4700 legacy=0x7f60c4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:30.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.094+0000 7f60c88b3640 1 -- 172.21.15.169:0/2434008311 shutdown_connections 2024-03-25T17:56:30.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.094+0000 7f60c88b3640 1 -- 172.21.15.169:0/2434008311 >> 172.21.15.169:0/2434008311 conn(0x7f60c40668e0 msgr2=0x7f60c4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:30.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.094+0000 7f60c88b3640 1 -- 172.21.15.169:0/2434008311 shutdown_connections 2024-03-25T17:56:30.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.094+0000 7f60c88b3640 1 -- 172.21.15.169:0/2434008311 wait complete. 2024-03-25T17:56:30.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.095+0000 7f60c88b3640 10 monclient: build_initial_monmap 2024-03-25T17:56:30.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.095+0000 7f60c88b3640 10 monclient: monmap: 2024-03-25T17:56:30.099 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T17:56:30.100 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:56:30.100 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-25T17:56:30.100 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-25T17:56:30.100 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:30.100 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-25T17:56:30.100 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:30.100 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:56:30.100 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:56:30.100 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:30.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.095+0000 7f60c88b3640 1 librados: starting msgr at 2024-03-25T17:56:30.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.095+0000 7f60c88b3640 1 librados: starting objecter 2024-03-25T17:56:30.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.095+0000 7f60c88b3640 1 Processor -- start 2024-03-25T17:56:30.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.095+0000 7f60c88b3640 1 -- start start 2024-03-25T17:56:30.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.095+0000 7f60c88b3640 1 librados: setting wanted keys 2024-03-25T17:56:30.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.095+0000 7f60c88b3640 1 librados: calling monclient init 2024-03-25T17:56:30.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.095+0000 7f60c88b3640 10 monclient: init 2024-03-25T17:56:30.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.096+0000 7f60c88b3640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:30.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.096+0000 7f60c88b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:56:30.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.096+0000 7f60c88b3640 10 monclient(hunting): picked mon.noname-b con 0x7f60c40a96b0 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:30.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.096+0000 7f60c88b3640 10 monclient(hunting): picked mon.noname-a con 0x7f60c40b4700 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:30.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.096+0000 7f60c88b3640 10 monclient(hunting): picked mon.noname-c con 0x7f60c4142a20 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:30.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.096+0000 7f60c88b3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c413e8d0 con 0x7f60c40b4700 2024-03-25T17:56:30.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.096+0000 7f60c88b3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c4133860 con 0x7f60c40a96b0 2024-03-25T17:56:30.101 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.096+0000 7f60c88b3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c4133b40 con 0x7f60c4142a20 2024-03-25T17:56:30.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.096+0000 7f60c88b3640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:30.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.096+0000 7f60c88b3640 10 monclient(hunting): authenticate will time out at 6905.381807s 2024-03-25T17:56:30.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.097+0000 7f60c259c640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f60c40a96b0 0x7f60c414d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.169:33064/0 (socket says 172.21.15.169:33064) 2024-03-25T17:56:30.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.097+0000 7f60c259c640 1 -- 172.21.15.169:0/3322872308 learned_addr learned my addr 172.21.15.169:0/3322872308 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T17:56:30.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.097+0000 7f60baffd640 10 client.?.objecter ms_handle_connect 0x7f60c40a96b0 2024-03-25T17:56:30.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.097+0000 7f60baffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:30.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.097+0000 7f60baffd640 10 client.?.objecter ms_handle_connect 0x7f60c40b4700 2024-03-25T17:56:30.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.097+0000 7f60baffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:30.102 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.097+0000 7f60baffd640 10 client.?.objecter ms_handle_connect 0x7f60c4142a20 2024-03-25T17:56:30.103 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.097+0000 7f60baffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:30.106 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867207194 0 0) 0x7f60c413e8d0 con 0x7f60c40b4700 2024-03-25T17:56:30.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 10 monclient(hunting): _negotiate 2024-03-25T17:56:30.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:30.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:30.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 10 monclient(hunting): my global_id is 4155 2024-03-25T17:56:30.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60a4004eb0 con 0x7f60c40b4700 2024-03-25T17:56:30.107 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998233539 0 0) 0x7f60c4133860 con 0x7f60c40a96b0 2024-03-25T17:56:30.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 10 monclient(hunting): _negotiate 2024-03-25T17:56:30.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:30.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:30.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 10 monclient(hunting): my global_id is 4156 2024-03-25T17:56:30.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c413e8d0 con 0x7f60c40a96b0 2024-03-25T17:56:30.108 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223124121 0 0) 0x7f60c4133b40 con 0x7f60c4142a20 2024-03-25T17:56:30.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 10 monclient(hunting): _negotiate 2024-03-25T17:56:30.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:30.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:30.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 10 monclient(hunting): my global_id is 4157 2024-03-25T17:56:30.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.099+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c4133860 con 0x7f60c4142a20 2024-03-25T17:56:30.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.104+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3573481299 0 0) 0x7f60a4004eb0 con 0x7f60c40b4700 2024-03-25T17:56:30.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.104+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60c4133b40 con 0x7f60c40b4700 2024-03-25T17:56:30.109 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.104+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3659309946 0 0) 0x7f60c413e8d0 con 0x7f60c40a96b0 2024-03-25T17:56:30.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.104+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60a4004eb0 con 0x7f60c40a96b0 2024-03-25T17:56:30.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.104+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f60ac0012f0 con 0x7f60c40b4700 2024-03-25T17:56:30.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.104+0000 7f60baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:30.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.104+0000 7f60baffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:56:30.110 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.104+0000 7f60baffd640 10 monclient(hunting): dump: 2024-03-25T17:56:30.110 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:30.110 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:30.110 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.111 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.111 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:30.111 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:30.111 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:30.111 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:30.111 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:30.111 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:30.111 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.104+0000 7f60baffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:56:30.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.104+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f60b4001630 con 0x7f60c40a96b0 2024-03-25T17:56:30.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.104+0000 7f60baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:30.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.104+0000 7f60baffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:56:30.112 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.104+0000 7f60baffd640 10 monclient(hunting): dump: 2024-03-25T17:56:30.112 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:30.112 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:30.112 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.113 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.113 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:30.113 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:30.113 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:30.113 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:30.113 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:30.113 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:30.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.105+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2879875959 0 0) 0x7f60c4133860 con 0x7f60c4142a20 2024-03-25T17:56:30.113 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.105+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60c413e8d0 con 0x7f60c4142a20 2024-03-25T17:56:30.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.105+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f60bc001bd0 con 0x7f60c4142a20 2024-03-25T17:56:30.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.105+0000 7f60baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:30.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.105+0000 7f60baffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:56:30.114 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.105+0000 7f60baffd640 10 monclient(hunting): dump: 2024-03-25T17:56:30.114 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:30.114 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:30.114 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.114 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.114 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:30.115 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:30.115 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:30.115 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:30.115 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:30.115 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:30.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3057542693 0 0) 0x7f60a4004eb0 con 0x7f60c40a96b0 2024-03-25T17:56:30.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 >> v1:172.21.15.18:6790/0 conn(0x7f60c4142a20 legacy=0x7f60c4134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:30.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 >> v1:172.21.15.18:6789/0 conn(0x7f60c40b4700 legacy=0x7f60c414ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:30.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60baffd640 10 monclient: _finish_hunting 0 2024-03-25T17:56:30.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60baffd640 1 monclient: found mon.b 2024-03-25T17:56:30.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:30.119 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60baffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:30.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c413edb0 con 0x7f60c40a96b0 2024-03-25T17:56:30.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60baffd640 10 monclient: _finish_auth 0 2024-03-25T17:56:30.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60baffd640 10 monclient: _check_auth_tickets 2024-03-25T17:56:30.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:30.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60c88b3640 5 monclient: authenticate success, global_id 4156 2024-03-25T17:56:30.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60c88b3640 10 monclient: _renew_subs 2024-03-25T17:56:30.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60c88b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:30.120 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60c88b3640 1 -- 172.21.15.169:0/3322872308 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c414e4d0 con 0x7f60c40a96b0 2024-03-25T17:56:30.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60c88b3640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:56:30.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60c88b3640 10 monclient: _renew_subs 2024-03-25T17:56:30.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60c88b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:30.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60c88b3640 1 -- 172.21.15.169:0/3322872308 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c4131f10 con 0x7f60c40a96b0 2024-03-25T17:56:30.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.107+0000 7f60c88b3640 1 librados: init done 2024-03-25T17:56:30.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.108+0000 7f60c88b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:30.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.108+0000 7f60c88b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:30.121 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.108+0000 7f60c88b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:30.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.108+0000 7f60c88b3640 1 -- 172.21.15.169:0/3322872308 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c404f730 con 0x7f60c40a96b0 2024-03-25T17:56:30.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.111+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60b4001ca0 con 0x7f60c40a96b0 2024-03-25T17:56:30.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.111+0000 7f60baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:30.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.111+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f60b4003a40 con 0x7f60c40a96b0 2024-03-25T17:56:30.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.111+0000 7f60baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:30.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.111+0000 7f60baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:56:30.122 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.111+0000 7f60baffd640 10 monclient: dump: 2024-03-25T17:56:30.123 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:30.123 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:30.123 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.123 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.123 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:30.123 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:30.123 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:30.123 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:30.124 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:30.124 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:30.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.111+0000 7f60c159a640 4 set_mon_vals no callback set 2024-03-25T17:56:30.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.111+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f60b4003ff0 con 0x7f60c40a96b0 2024-03-25T17:56:30.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.111+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1262763026 0 0) 0x7f60b4006310 con 0x7f60c40a96b0 2024-03-25T17:56:30.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.111+0000 7f60baffd640 10 client.4156.objecter ms_dispatch 0x7f60c4000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T17:56:30.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.111+0000 7f60baffd640 3 client.4156.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T17:56:30.124 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.111+0000 7f60baffd640 3 client.4156.objecter handle_osd_map decoding full epoch 8 2024-03-25T17:56:30.125 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.111+0000 7f60baffd640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T17:56:30.187 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:30.186+0000 7fc44ee54040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T17:56:30.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.206+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f60b4006540 con 0x7f60c40a96b0 2024-03-25T17:56:30.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.206+0000 7f60baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:30.206 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.206+0000 7f60baffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:56:30.224 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:30.224+0000 7fb497bf1040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T17:56:30.278 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.278+0000 7f60c88b3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c3bd1d77-7390-4452-8980-b0ef85bc74a8", "id": 6}] 2024-03-25T17:56:30.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.278+0000 7f60c88b3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c3bd1d77-7390-4452-8980-b0ef85bc74a8", "id": 6}] 2024-03-25T17:56:30.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.278+0000 7f60c88b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:30.279 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.278+0000 7f60c88b3640 1 -- 172.21.15.169:0/3322872308 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c3bd1d77-7390-4452-8980-b0ef85bc74a8", "id": 6} v 0) v1 -- 0x7f60c413f950 con 0x7f60c40a96b0 2024-03-25T17:56:30.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.360+0000 7f60baffd640 1 -- 172.21.15.169:0/3322872308 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c3bd1d77-7390-4452-8980-b0ef85bc74a8", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 351362417 0 1113811849) 0x7f60b40043c0 con 0x7f60c40a96b0 2024-03-25T17:56:30.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.360+0000 7f60baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c3bd1d77-7390-4452-8980-b0ef85bc74a8", "id": 6}] 2024-03-25T17:56:30.360 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.360+0000 7f60baffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:56:30.361 INFO:teuthology.orchestra.run.smithi169.stdout:6 2024-03-25T17:56:30.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.361+0000 7f60c88b3640 10 librados: watch_flush enter 2024-03-25T17:56:30.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.361+0000 7f60c88b3640 10 librados: watch_flush exit 2024-03-25T17:56:30.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.361+0000 7f60c88b3640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T17:56:30.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.361+0000 7f60c88b3640 10 client.4156.objecter successfully canceled tick 2024-03-25T17:56:30.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.361+0000 7f60c88b3640 10 monclient: shutdown 2024-03-25T17:56:30.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.361+0000 7f60c88b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:30.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.361+0000 7f60c88b3640 1 -- 172.21.15.169:0/3322872308 >> v1:172.21.15.169:6789/0 conn(0x7f60c40a96b0 legacy=0x7f60c414d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:30.362 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.361+0000 7f60c88b3640 1 -- 172.21.15.169:0/3322872308 shutdown_connections 2024-03-25T17:56:30.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.361+0000 7f60c88b3640 1 -- 172.21.15.169:0/3322872308 >> 172.21.15.169:0/3322872308 conn(0x7f60c40668e0 msgr2=0x7f60c4066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:30.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.361+0000 7f60c88b3640 1 -- 172.21.15.169:0/3322872308 shutdown_connections 2024-03-25T17:56:30.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.361+0000 7f60c88b3640 1 -- 172.21.15.169:0/3322872308 wait complete. 2024-03-25T17:56:30.363 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.361+0000 7f60c88b3640 1 librados: shutdown 2024-03-25T17:56:30.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 7a735ce9-3492-4d28-afa8-13314d24c12a 7 2024-03-25T17:56:30.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.473+0000 7f96b7b64640 10 monclient: get_monmap_and_config 2024-03-25T17:56:30.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.473+0000 7f96b7b64640 10 monclient: build_initial_monmap 2024-03-25T17:56:30.474 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.473+0000 7f96b7b64640 10 monclient: monmap: 2024-03-25T17:56:30.474 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T17:56:30.474 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:30.474 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:56:30.474538+0000 2024-03-25T17:56:30.474 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:56:30.474538+0000 2024-03-25T17:56:30.474 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:30.474 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:30.474 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:30.474 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:56:30.474 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:56:30.475 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:30.475 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.474+0000 7f96b7b64640 1 Processor -- start 2024-03-25T17:56:30.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.474+0000 7f96b7b64640 1 -- start start 2024-03-25T17:56:30.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.474+0000 7f96b7b64640 10 monclient: init 2024-03-25T17:56:30.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.474+0000 7f96b7b64640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:30.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.474+0000 7f96b7b64640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:56:30.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.474+0000 7f96b7b64640 10 monclient(hunting): picked mon.noname-a con 0x7f96b00b4700 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:30.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.474+0000 7f96b7b64640 10 monclient(hunting): picked mon.noname-b con 0x7f96b0142a20 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:30.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.474+0000 7f96b7b64640 10 monclient(hunting): picked mon.noname-c con 0x7f96b00a96b0 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:30.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.475+0000 7f96b7b64640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b00a9d60 con 0x7f96b00b4700 2024-03-25T17:56:30.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.475+0000 7f96b7b64640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b013ff10 con 0x7f96b00a96b0 2024-03-25T17:56:30.476 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.475+0000 7f96b7b64640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b00aa1f0 con 0x7f96b0142a20 2024-03-25T17:56:30.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.475+0000 7f96b7b64640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:30.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.475+0000 7f96b7b64640 10 monclient(hunting): authenticate will time out at 6905.760412s 2024-03-25T17:56:30.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.475+0000 7f96b5900640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f96b00b4700 0x7f96b0142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.169:34774/0 (socket says 172.21.15.169:34774) 2024-03-25T17:56:30.477 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.475+0000 7f96b5900640 1 -- 172.21.15.169:0/2542963624 learned_addr learned my addr 172.21.15.169:0/2542963624 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T17:56:30.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 1 -- 172.21.15.169:0/2542963624 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502905599 0 0) 0x7f96b00a9d60 con 0x7f96b00b4700 2024-03-25T17:56:30.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 10 monclient(hunting): _negotiate 2024-03-25T17:56:30.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:30.478 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:30.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 10 monclient(hunting): my global_id is 4158 2024-03-25T17:56:30.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 1 -- 172.21.15.169:0/2542963624 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f969c003520 con 0x7f96b00b4700 2024-03-25T17:56:30.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 1 -- 172.21.15.169:0/2542963624 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285070846 0 0) 0x7f96b013ff10 con 0x7f96b00a96b0 2024-03-25T17:56:30.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 10 monclient(hunting): _negotiate 2024-03-25T17:56:30.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:30.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:30.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 10 monclient(hunting): my global_id is 4159 2024-03-25T17:56:30.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 1 -- 172.21.15.169:0/2542963624 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96b00a9d60 con 0x7f96b00a96b0 2024-03-25T17:56:30.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 1 -- 172.21.15.169:0/2542963624 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235860326 0 0) 0x7f96b00aa1f0 con 0x7f96b0142a20 2024-03-25T17:56:30.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 10 monclient(hunting): _negotiate 2024-03-25T17:56:30.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:30.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:30.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 10 monclient(hunting): my global_id is 4160 2024-03-25T17:56:30.479 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.478+0000 7f96b48fe640 1 -- 172.21.15.169:0/2542963624 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96b013ff10 con 0x7f96b0142a20 2024-03-25T17:56:30.481 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:30.480+0000 7fc44ee54040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T17:56:30.481 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.481+0000 7f96b48fe640 1 -- 172.21.15.169:0/2542963624 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2022255151 0 0) 0x7f969c003520 con 0x7f96b00b4700 2024-03-25T17:56:30.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.481+0000 7f96b48fe640 1 -- 172.21.15.169:0/2542963624 >> v1:172.21.15.18:6790/0 conn(0x7f96b0142a20 legacy=0x7f96b013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:30.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.481+0000 7f96b48fe640 1 -- 172.21.15.169:0/2542963624 >> v1:172.21.15.169:6789/0 conn(0x7f96b00a96b0 legacy=0x7f96b013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:30.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.481+0000 7f96b48fe640 10 monclient: _finish_hunting 0 2024-03-25T17:56:30.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.481+0000 7f96b48fe640 1 monclient: found mon.noname-a 2024-03-25T17:56:30.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.481+0000 7f96b48fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:30.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.481+0000 7f96b48fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T17:56:30.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.481+0000 7f96b48fe640 1 -- 172.21.15.169:0/2542963624 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b00b4ae0 con 0x7f96b00b4700 2024-03-25T17:56:30.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.481+0000 7f96b48fe640 10 monclient: _finish_auth 0 2024-03-25T17:56:30.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.481+0000 7f96b48fe640 10 monclient: _check_auth_tickets 2024-03-25T17:56:30.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.481+0000 7f96b48fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:30.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.481+0000 7f96b48fe640 1 -- 172.21.15.169:0/2542963624 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f96ac001690 con 0x7f96b00b4700 2024-03-25T17:56:30.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.481+0000 7f96b48fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:30.482 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.481+0000 7f96b48fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:56:30.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.481+0000 7f96b48fe640 10 monclient: dump: 2024-03-25T17:56:30.483 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:30.483 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:30.483 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.483 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.483 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:30.483 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:30.483 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:30.483 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:30.483 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:30.483 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:30.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.481+0000 7f96b7b64640 5 monclient: authenticate success, global_id 4158 2024-03-25T17:56:30.483 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.481+0000 7f96b7b64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:30.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.483+0000 7f96b48fe640 1 -- 172.21.15.169:0/2542963624 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96ac0017f0 con 0x7f96b00b4700 2024-03-25T17:56:30.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.483+0000 7f96b48fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:30.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.483+0000 7f96b48fe640 1 -- 172.21.15.169:0/2542963624 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f96ac001c00 con 0x7f96b00b4700 2024-03-25T17:56:30.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.483+0000 7f96b7b64640 10 monclient: get_monmap_and_config success 2024-03-25T17:56:30.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.483+0000 7f96b7b64640 4 set_mon_vals no callback set 2024-03-25T17:56:30.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.483+0000 7f96b48fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:30.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.483+0000 7f96b48fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:30.486 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.483+0000 7f96b48fe640 10 monclient: dump: 2024-03-25T17:56:30.486 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:30.486 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:30.486 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.487 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.487 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:30.487 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:30.487 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:30.487 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:30.487 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:30.487 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:30.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.484+0000 7f96b7b64640 10 monclient: shutdown 2024-03-25T17:56:30.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.484+0000 7f96b7b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:30.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.484+0000 7f96b7b64640 1 -- 172.21.15.169:0/2542963624 >> v1:172.21.15.18:6789/0 conn(0x7f96b00b4700 legacy=0x7f96b0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:30.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.484+0000 7f96b7b64640 1 -- 172.21.15.169:0/2542963624 shutdown_connections 2024-03-25T17:56:30.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.484+0000 7f96b7b64640 1 -- 172.21.15.169:0/2542963624 >> 172.21.15.169:0/2542963624 conn(0x7f96b00668e0 msgr2=0x7f96b0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:30.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.484+0000 7f96b7b64640 1 -- 172.21.15.169:0/2542963624 shutdown_connections 2024-03-25T17:56:30.487 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.484+0000 7f96b7b64640 1 -- 172.21.15.169:0/2542963624 wait complete. 2024-03-25T17:56:30.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.484+0000 7f96b7b64640 10 monclient: build_initial_monmap 2024-03-25T17:56:30.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.484+0000 7f96b7b64640 10 monclient: monmap: 2024-03-25T17:56:30.488 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 0 2024-03-25T17:56:30.488 INFO:teuthology.orchestra.run.smithi169.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:56:30.488 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 0.000000 2024-03-25T17:56:30.488 INFO:teuthology.orchestra.run.smithi169.stderr:created 0.000000 2024-03-25T17:56:30.488 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:30.488 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 1 2024-03-25T17:56:30.488 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:30.488 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:56:30.488 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:56:30.488 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:30.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.485+0000 7f96b7b64640 1 librados: starting msgr at 2024-03-25T17:56:30.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.485+0000 7f96b7b64640 1 librados: starting objecter 2024-03-25T17:56:30.488 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.485+0000 7f96b7b64640 1 Processor -- start 2024-03-25T17:56:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.485+0000 7f96b7b64640 1 -- start start 2024-03-25T17:56:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.485+0000 7f96b7b64640 1 librados: setting wanted keys 2024-03-25T17:56:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.485+0000 7f96b7b64640 1 librados: calling monclient init 2024-03-25T17:56:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.485+0000 7f96b7b64640 10 monclient: init 2024-03-25T17:56:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.485+0000 7f96b7b64640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.485+0000 7f96b7b64640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:56:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.485+0000 7f96b7b64640 10 monclient(hunting): picked mon.noname-b con 0x7f96b00a96b0 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.485+0000 7f96b7b64640 10 monclient(hunting): picked mon.noname-c con 0x7f96b00b4700 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.485+0000 7f96b7b64640 10 monclient(hunting): picked mon.noname-a con 0x7f96b0142a20 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.485+0000 7f96b7b64640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b013a4d0 con 0x7f96b0142a20 2024-03-25T17:56:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.485+0000 7f96b7b64640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b01da880 con 0x7f96b00a96b0 2024-03-25T17:56:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.485+0000 7f96b7b64640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b0139240 con 0x7f96b00b4700 2024-03-25T17:56:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.485+0000 7f96b7b64640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:30.489 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.485+0000 7f96b7b64640 10 monclient(hunting): authenticate will time out at 6905.771116s 2024-03-25T17:56:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.486+0000 7f96b6101640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f96b0142a20 0x7f96b01e1bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.169:34786/0 (socket says 172.21.15.169:34786) 2024-03-25T17:56:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.486+0000 7f96b6101640 1 -- 172.21.15.169:0/2525340621 learned_addr learned my addr 172.21.15.169:0/2525340621 (peer_addr_for_me v1:172.21.15.169:0/0) 2024-03-25T17:56:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.486+0000 7f96a67fc640 10 client.?.objecter ms_handle_connect 0x7f96b0142a20 2024-03-25T17:56:30.490 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.486+0000 7f96a67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.486+0000 7f96a67fc640 10 client.?.objecter ms_handle_connect 0x7f96b00b4700 2024-03-25T17:56:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.486+0000 7f96a67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.486+0000 7f96a67fc640 10 client.?.objecter ms_handle_connect 0x7f96b00a96b0 2024-03-25T17:56:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.486+0000 7f96a67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.488+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168618634 0 0) 0x7f96b013a4d0 con 0x7f96b0142a20 2024-03-25T17:56:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.488+0000 7f96a67fc640 10 monclient(hunting): _negotiate 2024-03-25T17:56:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.488+0000 7f96a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.488+0000 7f96a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:30.491 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.488+0000 7f96a67fc640 10 monclient(hunting): my global_id is 4161 2024-03-25T17:56:30.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.488+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9694004eb0 con 0x7f96b0142a20 2024-03-25T17:56:30.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.488+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137918744 0 0) 0x7f96b01da880 con 0x7f96b00a96b0 2024-03-25T17:56:30.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.488+0000 7f96a67fc640 10 monclient(hunting): _negotiate 2024-03-25T17:56:30.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.488+0000 7f96a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:30.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.488+0000 7f96a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:30.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.488+0000 7f96a67fc640 10 monclient(hunting): my global_id is 4162 2024-03-25T17:56:30.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.488+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96b013a4d0 con 0x7f96b00a96b0 2024-03-25T17:56:30.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.489+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040875793 0 0) 0x7f96b0139240 con 0x7f96b00b4700 2024-03-25T17:56:30.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.489+0000 7f96a67fc640 10 monclient(hunting): _negotiate 2024-03-25T17:56:30.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.489+0000 7f96a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:30.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.489+0000 7f96a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:30.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.489+0000 7f96a67fc640 10 monclient(hunting): my global_id is 4163 2024-03-25T17:56:30.492 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.489+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96b01da880 con 0x7f96b00b4700 2024-03-25T17:56:30.493 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.493+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2451744982 0 0) 0x7f9694004eb0 con 0x7f96b0142a20 2024-03-25T17:56:30.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.493+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96b0139240 con 0x7f96b0142a20 2024-03-25T17:56:30.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.493+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f96a8001c90 con 0x7f96b0142a20 2024-03-25T17:56:30.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.493+0000 7f96a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:30.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.493+0000 7f96a67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:56:30.494 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.493+0000 7f96a67fc640 10 monclient(hunting): dump: 2024-03-25T17:56:30.494 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:30.494 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:30.494 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.494 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.494 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:30.494 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:30.494 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:30.494 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:30.495 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:30.495 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:30.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.493+0000 7f96a67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:56:30.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.493+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2308686659 0 0) 0x7f96b013a4d0 con 0x7f96b00a96b0 2024-03-25T17:56:30.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.493+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9694004eb0 con 0x7f96b00a96b0 2024-03-25T17:56:30.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.493+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f96ac001b00 con 0x7f96b00a96b0 2024-03-25T17:56:30.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.493+0000 7f96a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:30.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.493+0000 7f96a67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:56:30.495 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.493+0000 7f96a67fc640 10 monclient(hunting): dump: 2024-03-25T17:56:30.495 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:30.495 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:30.495 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.495 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.495 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:30.495 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:30.496 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:30.496 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:30.496 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:30.496 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:30.497 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.495+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4272119842 0 0) 0x7f96b01da880 con 0x7f96b00b4700 2024-03-25T17:56:30.497 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.495+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96b013a4d0 con 0x7f96b00b4700 2024-03-25T17:56:30.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.495+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f96a0001830 con 0x7f96b00b4700 2024-03-25T17:56:30.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.495+0000 7f96a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:30.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.495+0000 7f96a67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:56:30.498 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.495+0000 7f96a67fc640 10 monclient(hunting): dump: 2024-03-25T17:56:30.498 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:30.498 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:30.498 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.498 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.498 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:30.498 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:30.498 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:30.498 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:30.498 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:30.499 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:30.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2237183110 0 0) 0x7f96b0139240 con 0x7f96b0142a20 2024-03-25T17:56:30.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 >> v1:172.21.15.18:6790/0 conn(0x7f96b00b4700 legacy=0x7f96b01db040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:30.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 >> v1:172.21.15.169:6789/0 conn(0x7f96b00a96b0 legacy=0x7f96b0138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:30.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96a67fc640 10 monclient: _finish_hunting 0 2024-03-25T17:56:30.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96a67fc640 1 monclient: found mon.a 2024-03-25T17:56:30.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:30.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96a67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:30.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b01d4fe0 con 0x7f96b0142a20 2024-03-25T17:56:30.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96a67fc640 10 monclient: _finish_auth 0 2024-03-25T17:56:30.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96a67fc640 10 monclient: _check_auth_tickets 2024-03-25T17:56:30.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:30.499 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96b7b64640 5 monclient: authenticate success, global_id 4161 2024-03-25T17:56:30.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96b7b64640 10 monclient: _renew_subs 2024-03-25T17:56:30.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96b7b64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:30.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96b7b64640 1 -- 172.21.15.169:0/2525340621 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b01dab20 con 0x7f96b0142a20 2024-03-25T17:56:30.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96b7b64640 10 client.4161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:56:30.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96b7b64640 10 monclient: _renew_subs 2024-03-25T17:56:30.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96b7b64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:30.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96b7b64640 1 -- 172.21.15.169:0/2525340621 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b01db800 con 0x7f96b0142a20 2024-03-25T17:56:30.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.496+0000 7f96b7b64640 1 librados: init done 2024-03-25T17:56:30.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.497+0000 7f96b7b64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:30.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.497+0000 7f96b7b64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:30.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.497+0000 7f96b7b64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:30.500 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.497+0000 7f96b7b64640 1 -- 172.21.15.169:0/2525340621 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b004f730 con 0x7f96b0142a20 2024-03-25T17:56:30.518 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:30.518+0000 7fb497bf1040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T17:56:30.538 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:30.538+0000 7fc44ee54040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T17:56:30.569 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:30.569+0000 7fb497bf1040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T17:56:30.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.603+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96a8003970 con 0x7f96b0142a20 2024-03-25T17:56:30.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.603+0000 7f96a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:30.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.603+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f96a8003d00 con 0x7f96b0142a20 2024-03-25T17:56:30.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.603+0000 7f96a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:30.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.603+0000 7f96a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:30.604 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.603+0000 7f96a67fc640 10 monclient: dump: 2024-03-25T17:56:30.604 INFO:teuthology.orchestra.run.smithi169.stderr:epoch 1 2024-03-25T17:56:30.604 INFO:teuthology.orchestra.run.smithi169.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:30.604 INFO:teuthology.orchestra.run.smithi169.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.604 INFO:teuthology.orchestra.run.smithi169.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:30.604 INFO:teuthology.orchestra.run.smithi169.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:30.605 INFO:teuthology.orchestra.run.smithi169.stderr:election_strategy: 3 2024-03-25T17:56:30.605 INFO:teuthology.orchestra.run.smithi169.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:30.605 INFO:teuthology.orchestra.run.smithi169.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:30.605 INFO:teuthology.orchestra.run.smithi169.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:30.605 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T17:56:30.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.603+0000 7f96a7fff640 4 set_mon_vals no callback set 2024-03-25T17:56:30.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.603+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f96a80042b0 con 0x7f96b0142a20 2024-03-25T17:56:30.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.603+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1227329192 0 0) 0x7f96a8006b10 con 0x7f96b0142a20 2024-03-25T17:56:30.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.603+0000 7f96a67fc640 10 client.4161.objecter ms_dispatch 0x7f96b0000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T17:56:30.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.603+0000 7f96a67fc640 3 client.4161.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T17:56:30.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.603+0000 7f96a67fc640 3 client.4161.objecter handle_osd_map decoding full epoch 9 2024-03-25T17:56:30.605 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.603+0000 7f96a67fc640 20 client.4161.objecter dump_active .. 0 homeless 2024-03-25T17:56:30.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.604+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f96a8004560 con 0x7f96b0142a20 2024-03-25T17:56:30.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.604+0000 7f96a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:30.606 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.604+0000 7f96a67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:56:30.660 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:30.659+0000 7fc44ee54040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T17:56:30.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.676+0000 7f96b7b64640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7a735ce9-3492-4d28-afa8-13314d24c12a", "id": 7}] 2024-03-25T17:56:30.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.676+0000 7f96b7b64640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7a735ce9-3492-4d28-afa8-13314d24c12a", "id": 7}] 2024-03-25T17:56:30.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.676+0000 7f96b7b64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:30.677 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.676+0000 7f96b7b64640 1 -- 172.21.15.169:0/2525340621 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7a735ce9-3492-4d28-afa8-13314d24c12a", "id": 7} v 0) v1 -- 0x7f96b0069f30 con 0x7f96b0142a20 2024-03-25T17:56:30.688 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-03-25T17:56:30.688+0000 7fb497bf1040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T17:56:30.904 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:30.904+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 2167572322 0 0) 0x7f96a8006560 con 0x7f96b0142a20 2024-03-25T17:56:31.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:31.146+0000 7f96a67fc640 1 -- 172.21.15.169:0/2525340621 <== mon.0 v1:172.21.15.18:6789/0 11 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7a735ce9-3492-4d28-afa8-13314d24c12a", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 4038642361 0 1371665406) 0x7f96a80162a0 con 0x7f96b0142a20 2024-03-25T17:56:31.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:31.146+0000 7f96a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7a735ce9-3492-4d28-afa8-13314d24c12a", "id": 7}] 2024-03-25T17:56:31.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:31.146+0000 7f96a67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:56:31.147 INFO:teuthology.orchestra.run.smithi169.stdout:7 2024-03-25T17:56:31.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:31.147+0000 7f96b7b64640 10 librados: watch_flush enter 2024-03-25T17:56:31.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:31.147+0000 7f96b7b64640 10 librados: watch_flush exit 2024-03-25T17:56:31.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:31.147+0000 7f96b7b64640 20 client.4161.objecter shutdown clearing up homeless session... 2024-03-25T17:56:31.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:31.147+0000 7f96b7b64640 10 client.4161.objecter successfully canceled tick 2024-03-25T17:56:31.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:31.147+0000 7f96b7b64640 10 monclient: shutdown 2024-03-25T17:56:31.148 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:31.147+0000 7f96b7b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:31.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:31.147+0000 7f96b7b64640 1 -- 172.21.15.169:0/2525340621 >> v1:172.21.15.18:6789/0 conn(0x7f96b0142a20 legacy=0x7f96b01e1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:31.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:31.147+0000 7f96b7b64640 1 -- 172.21.15.169:0/2525340621 shutdown_connections 2024-03-25T17:56:31.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:31.147+0000 7f96b7b64640 1 -- 172.21.15.169:0/2525340621 >> 172.21.15.169:0/2525340621 conn(0x7f96b00668e0 msgr2=0x7f96b013e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:31.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:31.147+0000 7f96b7b64640 1 -- 172.21.15.169:0/2525340621 shutdown_connections 2024-03-25T17:56:31.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:31.147+0000 7f96b7b64640 1 -- 172.21.15.169:0/2525340621 wait complete. 2024-03-25T17:56:31.149 INFO:teuthology.orchestra.run.smithi169.stderr:2024-03-25T17:56:31.147+0000 7f96b7b64640 1 librados: shutdown 2024-03-25T17:56:31.159 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:56:31.159 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T17:56:31.160 DEBUG:teuthology.orchestra.run.smithi018:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T17:56:31.162 INFO:tasks.ceph.osd.0:Started 2024-03-25T17:56:31.162 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:56:31.162 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T17:56:31.162 DEBUG:teuthology.orchestra.run.smithi018:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T17:56:31.165 INFO:tasks.ceph.osd.1:Started 2024-03-25T17:56:31.165 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:56:31.165 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T17:56:31.165 DEBUG:teuthology.orchestra.run.smithi018:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T17:56:31.167 INFO:tasks.ceph.osd.2:Started 2024-03-25T17:56:31.168 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:56:31.168 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T17:56:31.168 DEBUG:teuthology.orchestra.run.smithi018:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T17:56:31.170 INFO:tasks.ceph.osd.3:Started 2024-03-25T17:56:31.170 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:56:31.171 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T17:56:31.171 DEBUG:teuthology.orchestra.run.smithi169:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T17:56:31.173 INFO:tasks.ceph.osd.4:Started 2024-03-25T17:56:31.173 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:56:31.173 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T17:56:31.173 DEBUG:teuthology.orchestra.run.smithi169:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T17:56:31.175 INFO:tasks.ceph.osd.5:Started 2024-03-25T17:56:31.175 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:56:31.175 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T17:56:31.175 DEBUG:teuthology.orchestra.run.smithi169:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T17:56:31.177 INFO:tasks.ceph.osd.6:Started 2024-03-25T17:56:31.177 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:56:31.177 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T17:56:31.178 DEBUG:teuthology.orchestra.run.smithi169:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T17:56:31.180 INFO:tasks.ceph.osd.7:Started 2024-03-25T17:56:31.180 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T17:56:31.339 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.338+0000 7fc5d364e640 10 monclient: get_monmap_and_config 2024-03-25T17:56:31.339 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.338+0000 7fc5d364e640 10 monclient: build_initial_monmap 2024-03-25T17:56:31.339 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.338+0000 7fc5d364e640 10 monclient: monmap: 2024-03-25T17:56:31.339 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 0 2024-03-25T17:56:31.339 INFO:teuthology.orchestra.run.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:31.339 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T17:56:31.340033+0000 2024-03-25T17:56:31.339 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T17:56:31.340033+0000 2024-03-25T17:56:31.340 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:31.340 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:31.340 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:31.340 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:56:31.340 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:56:31.340 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T17:56:31.341 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.339+0000 7fc5d364e640 1 Processor -- start 2024-03-25T17:56:31.341 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.339+0000 7fc5d364e640 1 -- start start 2024-03-25T17:56:31.342 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.339+0000 7fc5d364e640 10 monclient: init 2024-03-25T17:56:31.342 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.340+0000 7fc5d364e640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:31.342 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.340+0000 7fc5d364e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:56:31.342 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.340+0000 7fc5d364e640 10 monclient(hunting): picked mon.noname-b con 0x7fc5cc0b4700 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:31.342 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.340+0000 7fc5d364e640 10 monclient(hunting): picked mon.noname-a con 0x7fc5cc142a20 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:31.342 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.340+0000 7fc5d364e640 10 monclient(hunting): picked mon.noname-c con 0x7fc5cc0a96b0 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:31.342 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.340+0000 7fc5d364e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc0a9d60 con 0x7fc5cc142a20 2024-03-25T17:56:31.342 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.340+0000 7fc5d364e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc13ff10 con 0x7fc5cc0a96b0 2024-03-25T17:56:31.342 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.340+0000 7fc5d364e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc0aa1f0 con 0x7fc5cc0b4700 2024-03-25T17:56:31.342 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.340+0000 7fc5d364e640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:31.342 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.340+0000 7fc5d364e640 10 monclient(hunting): authenticate will time out at 6919.919455s 2024-03-25T17:56:31.347 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.346+0000 7fc5d0be9640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fc5cc142a20 0x7fc5cc13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:33944/0 (socket says 172.21.15.18:33944) 2024-03-25T17:56:31.347 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.347+0000 7fc5d0be9640 1 -- 172.21.15.18:0/1834262893 learned_addr learned my addr 172.21.15.18:0/1834262893 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:56:31.358 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.357+0000 7fc5c3fff640 1 -- 172.21.15.18:0/1834262893 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516459886 0 0) 0x7fc5cc0aa1f0 con 0x7fc5cc0b4700 2024-03-25T17:56:31.358 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.357+0000 7fc5c3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:56:31.358 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.357+0000 7fc5c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:31.358 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.357+0000 7fc5c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:31.359 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.357+0000 7fc5c3fff640 10 monclient(hunting): my global_id is 4178 2024-03-25T17:56:31.359 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.357+0000 7fc5c3fff640 1 -- 172.21.15.18:0/1834262893 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b8003520 con 0x7fc5cc0b4700 2024-03-25T17:56:31.359 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.357+0000 7fc5c3fff640 1 -- 172.21.15.18:0/1834262893 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687933764 0 0) 0x7fc5cc0a9d60 con 0x7fc5cc142a20 2024-03-25T17:56:31.359 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.358+0000 7fc5c3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:56:31.359 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.358+0000 7fc5c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:31.359 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.358+0000 7fc5c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:31.359 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.358+0000 7fc5c3fff640 10 monclient(hunting): my global_id is 4170 2024-03-25T17:56:31.359 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.358+0000 7fc5c3fff640 1 -- 172.21.15.18:0/1834262893 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5cc0aa1f0 con 0x7fc5cc142a20 2024-03-25T17:56:31.362 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.361+0000 7fc5c3fff640 1 -- 172.21.15.18:0/1834262893 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 783139492 0 0) 0x7fc5b8003520 con 0x7fc5cc0b4700 2024-03-25T17:56:31.362 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.361+0000 7fc5c3fff640 1 -- 172.21.15.18:0/1834262893 >> v1:172.21.15.169:6789/0 conn(0x7fc5cc0a96b0 legacy=0x7fc5cc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:56:31.362 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.361+0000 7fc5c3fff640 1 -- 172.21.15.18:0/1834262893 >> v1:172.21.15.18:6789/0 conn(0x7fc5cc142a20 legacy=0x7fc5cc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:31.362 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.361+0000 7fc5c3fff640 10 monclient: _finish_hunting 0 2024-03-25T17:56:31.362 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.361+0000 7fc5c3fff640 1 monclient: found mon.noname-b 2024-03-25T17:56:31.362 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.361+0000 7fc5c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:31.362 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.361+0000 7fc5c3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T17:56:31.362 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.361+0000 7fc5c3fff640 1 -- 172.21.15.18:0/1834262893 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc0b4ae0 con 0x7fc5cc0b4700 2024-03-25T17:56:31.363 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.361+0000 7fc5c3fff640 10 monclient: _finish_auth 0 2024-03-25T17:56:31.363 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.361+0000 7fc5c3fff640 10 monclient: _check_auth_tickets 2024-03-25T17:56:31.363 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.361+0000 7fc5c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:31.363 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.361+0000 7fc5c3fff640 1 -- 172.21.15.18:0/1834262893 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc5bc001690 con 0x7fc5cc0b4700 2024-03-25T17:56:31.363 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.361+0000 7fc5c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:31.363 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.361+0000 7fc5c3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:56:31.363 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.361+0000 7fc5c3fff640 10 monclient: dump: 2024-03-25T17:56:31.363 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-25T17:56:31.363 INFO:teuthology.orchestra.run.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:31.363 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:31.363 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:31.363 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:31.363 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:31.364 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:31.364 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:31.364 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:31.364 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T17:56:31.364 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.361+0000 7fc5d364e640 5 monclient: authenticate success, global_id 4178 2024-03-25T17:56:31.364 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.361+0000 7fc5d364e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:31.366 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.366+0000 7fc5c3fff640 1 -- 172.21.15.18:0/1834262893 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5bc0017f0 con 0x7fc5cc0b4700 2024-03-25T17:56:31.367 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.366+0000 7fc5c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:31.367 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.366+0000 7fc5d364e640 10 monclient: get_monmap_and_config success 2024-03-25T17:56:31.367 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.366+0000 7fc5d364e640 4 set_mon_vals no callback set 2024-03-25T17:56:31.368 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.367+0000 7fc5d364e640 10 monclient: shutdown 2024-03-25T17:56:31.368 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.367+0000 7fc5d364e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:31.368 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.367+0000 7fc5d364e640 1 -- 172.21.15.18:0/1834262893 >> v1:172.21.15.18:6790/0 conn(0x7fc5cc0b4700 legacy=0x7fc5cc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:31.368 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.367+0000 7fc5d364e640 1 -- 172.21.15.18:0/1834262893 shutdown_connections 2024-03-25T17:56:31.368 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.367+0000 7fc5d364e640 1 -- 172.21.15.18:0/1834262893 >> 172.21.15.18:0/1834262893 conn(0x7fc5cc0668e0 msgr2=0x7fc5cc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:31.368 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.367+0000 7fc5d364e640 1 -- 172.21.15.18:0/1834262893 shutdown_connections 2024-03-25T17:56:31.368 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.367+0000 7fc5d364e640 1 -- 172.21.15.18:0/1834262893 wait complete. 2024-03-25T17:56:31.369 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.368+0000 7fc5d364e640 10 monclient: build_initial_monmap 2024-03-25T17:56:31.369 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.368+0000 7fc5d364e640 10 monclient: monmap: 2024-03-25T17:56:31.369 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 0 2024-03-25T17:56:31.369 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:56:31.369 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 0.000000 2024-03-25T17:56:31.369 INFO:teuthology.orchestra.run.smithi018.stderr:created 0.000000 2024-03-25T17:56:31.369 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:31.369 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-25T17:56:31.370 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:31.370 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:56:31.370 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:56:31.370 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T17:56:31.370 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.368+0000 7fc5d364e640 1 librados: starting msgr at 2024-03-25T17:56:31.370 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.368+0000 7fc5d364e640 1 librados: starting objecter 2024-03-25T17:56:31.370 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.369+0000 7fc5d364e640 1 Processor -- start 2024-03-25T17:56:31.370 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.369+0000 7fc5d364e640 1 -- start start 2024-03-25T17:56:31.370 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.369+0000 7fc5d364e640 1 librados: setting wanted keys 2024-03-25T17:56:31.370 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.369+0000 7fc5d364e640 1 librados: calling monclient init 2024-03-25T17:56:31.370 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.369+0000 7fc5d364e640 10 monclient: init 2024-03-25T17:56:31.372 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.369+0000 7fc5d364e640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:31.372 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.369+0000 7fc5d364e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:56:31.372 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.369+0000 7fc5d364e640 10 monclient(hunting): picked mon.noname-b con 0x7fc5cc0a96b0 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:31.372 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.370+0000 7fc5d364e640 10 monclient(hunting): picked mon.noname-c con 0x7fc5cc0b4700 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:31.372 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.370+0000 7fc5d364e640 10 monclient(hunting): picked mon.noname-a con 0x7fc5cc142a20 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:31.372 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.370+0000 7fc5d364e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc0b0180 con 0x7fc5cc142a20 2024-03-25T17:56:31.372 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.370+0000 7fc5d364e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc0afea0 con 0x7fc5cc0a96b0 2024-03-25T17:56:31.372 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.370+0000 7fc5d364e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5cc0abe20 con 0x7fc5cc0b4700 2024-03-25T17:56:31.373 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.370+0000 7fc5d364e640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:31.373 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.370+0000 7fc5d364e640 10 monclient(hunting): authenticate will time out at 6919.949287s 2024-03-25T17:56:31.373 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.370+0000 7fc5d0be9640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fc5cc0b4700 0x7fc5cc0ab710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:54260/0 (socket says 172.21.15.18:54260) 2024-03-25T17:56:31.373 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.370+0000 7fc5d0be9640 1 -- 172.21.15.18:0/3420859074 learned_addr learned my addr 172.21.15.18:0/3420859074 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:56:31.373 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.371+0000 7fc5c1ffb640 10 client.?.objecter ms_handle_connect 0x7fc5cc0b4700 2024-03-25T17:56:31.373 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.371+0000 7fc5c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:31.375 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.374+0000 7fc5c1ffb640 1 -- 172.21.15.18:0/3420859074 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173937768 0 0) 0x7fc5cc0abe20 con 0x7fc5cc0b4700 2024-03-25T17:56:31.375 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.374+0000 7fc5c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:56:31.376 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.374+0000 7fc5c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:31.376 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.374+0000 7fc5c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:31.376 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.374+0000 7fc5c1ffb640 10 monclient(hunting): my global_id is 4181 2024-03-25T17:56:31.376 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.374+0000 7fc5c1ffb640 1 -- 172.21.15.18:0/3420859074 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b0004eb0 con 0x7fc5cc0b4700 2024-03-25T17:56:31.383 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.382+0000 7fc5c1ffb640 10 client.?.objecter ms_handle_connect 0x7fc5cc142a20 2024-03-25T17:56:31.383 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.382+0000 7fc5c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:31.383 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.382+0000 7fc5c1ffb640 1 -- 172.21.15.18:0/3420859074 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2220035457 0 0) 0x7fc5b0004eb0 con 0x7fc5cc0b4700 2024-03-25T17:56:31.383 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.382+0000 7fc5c1ffb640 1 -- 172.21.15.18:0/3420859074 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5cc0abe20 con 0x7fc5cc0b4700 2024-03-25T17:56:31.383 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.382+0000 7fc5c1ffb640 1 -- 172.21.15.18:0/3420859074 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc5c8001970 con 0x7fc5cc0b4700 2024-03-25T17:56:31.383 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.382+0000 7fc5c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:31.383 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.382+0000 7fc5c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:56:31.383 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.382+0000 7fc5c1ffb640 10 monclient(hunting): dump: 2024-03-25T17:56:31.383 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-25T17:56:31.383 INFO:teuthology.orchestra.run.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:31.383 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:31.383 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:31.384 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:31.384 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:31.384 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:31.384 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:31.384 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:31.384 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T17:56:31.384 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.382+0000 7fc5c1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:56:31.390 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.387+0000 7fc5c1ffb640 1 -- 172.21.15.18:0/3420859074 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3550436408 0 0) 0x7fc5cc0abe20 con 0x7fc5cc0b4700 2024-03-25T17:56:31.390 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5c1ffb640 1 -- 172.21.15.18:0/3420859074 >> v1:172.21.15.169:6789/0 conn(0x7fc5cc0a96b0 legacy=0x7fc5cc0ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:31.390 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5c1ffb640 1 -- 172.21.15.18:0/3420859074 >> v1:172.21.15.18:6789/0 conn(0x7fc5cc142a20 legacy=0x7fc5cc1e63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:31.390 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5c1ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:56:31.391 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5c1ffb640 1 monclient: found mon.c 2024-03-25T17:56:31.391 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:31.391 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5c1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:56:31.391 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5c1ffb640 1 -- 172.21.15.18:0/3420859074 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc0adb20 con 0x7fc5cc0b4700 2024-03-25T17:56:31.391 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5c1ffb640 10 monclient: _finish_auth 0 2024-03-25T17:56:31.391 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5c1ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:56:31.391 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:31.391 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5d364e640 5 monclient: authenticate success, global_id 4181 2024-03-25T17:56:31.391 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5d364e640 10 monclient: _renew_subs 2024-03-25T17:56:31.391 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5d364e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:56:31.391 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5d364e640 1 -- 172.21.15.18:0/3420859074 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5cc13cc70 con 0x7fc5cc0b4700 2024-03-25T17:56:31.391 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5d364e640 10 client.4181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:56:31.391 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5d364e640 10 monclient: _renew_subs 2024-03-25T17:56:31.391 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5d364e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:56:31.392 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5d364e640 1 -- 172.21.15.18:0/3420859074 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5cc0b1800 con 0x7fc5cc0b4700 2024-03-25T17:56:31.392 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.388+0000 7fc5d364e640 1 librados: init done 2024-03-25T17:56:31.392 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.389+0000 7fc5d364e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:31.392 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.389+0000 7fc5d364e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:31.392 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.389+0000 7fc5d364e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:56:31.392 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.389+0000 7fc5d364e640 1 -- 172.21.15.18:0/3420859074 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5cc04f730 con 0x7fc5cc0b4700 2024-03-25T17:56:31.402 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.401+0000 7fc5c1ffb640 1 -- 172.21.15.18:0/3420859074 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5c8001c50 con 0x7fc5cc0b4700 2024-03-25T17:56:31.403 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.401+0000 7fc5c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:31.403 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.401+0000 7fc5c1ffb640 1 -- 172.21.15.18:0/3420859074 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc5c8004c30 con 0x7fc5cc0b4700 2024-03-25T17:56:31.403 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.401+0000 7fc5c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:31.403 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.401+0000 7fc5c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:56:31.403 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.401+0000 7fc5c1ffb640 10 monclient: dump: 2024-03-25T17:56:31.403 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-25T17:56:31.403 INFO:teuthology.orchestra.run.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:31.403 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:31.403 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:31.403 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:31.403 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:31.403 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:31.403 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:31.404 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:31.404 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T17:56:31.404 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.401+0000 7fc5c37fe640 4 set_mon_vals no callback set 2024-03-25T17:56:31.404 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.403+0000 7fc5c1ffb640 1 -- 172.21.15.18:0/3420859074 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 2167572322 0 0) 0x7fc5c800fd10 con 0x7fc5cc0b4700 2024-03-25T17:56:31.406 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.405+0000 7fc5c1ffb640 1 -- 172.21.15.18:0/3420859074 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fc5c8015770 con 0x7fc5cc0b4700 2024-03-25T17:56:31.406 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.405+0000 7fc5c1ffb640 10 client.4181.objecter ms_dispatch 0x7fc5cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:56:31.406 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.405+0000 7fc5c1ffb640 3 client.4181.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:56:31.406 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.405+0000 7fc5c1ffb640 3 client.4181.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:56:31.406 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.405+0000 7fc5c1ffb640 20 client.4181.objecter dump_active .. 0 homeless 2024-03-25T17:56:31.510 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.509+0000 7fc5c1ffb640 1 -- 172.21.15.18:0/3420859074 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc5c8014a30 con 0x7fc5cc0b4700 2024-03-25T17:56:31.510 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.509+0000 7fc5c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:31.511 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.509+0000 7fc5c1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:56:31.578 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.577+0000 7fc5d364e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:56:31.578 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.577+0000 7fc5d364e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:56:31.578 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.577+0000 7fc5d364e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:56:31.578 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.577+0000 7fc5d364e640 1 -- 172.21.15.18:0/3420859074 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5cc069f30 con 0x7fc5cc0b4700 2024-03-25T17:56:31.773 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.772+0000 7fc5c1ffb640 1 -- 172.21.15.18:0/3420859074 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fc5c80150c0 con 0x7fc5cc0b4700 2024-03-25T17:56:31.773 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.772+0000 7fc5c1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:56:31.773 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.772+0000 7fc5c1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:56:31.774 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-25T17:56:31.774 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:56:31.774 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.774+0000 7fc5d364e640 10 librados: watch_flush enter 2024-03-25T17:56:31.776 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.775+0000 7fc5d364e640 10 librados: watch_flush exit 2024-03-25T17:56:31.776 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.775+0000 7fc5d364e640 20 client.4181.objecter shutdown clearing up homeless session... 2024-03-25T17:56:31.776 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.775+0000 7fc5d364e640 10 client.4181.objecter successfully canceled tick 2024-03-25T17:56:31.776 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.775+0000 7fc5d364e640 10 monclient: shutdown 2024-03-25T17:56:31.776 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.775+0000 7fc5d364e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:31.776 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.775+0000 7fc5d364e640 1 -- 172.21.15.18:0/3420859074 >> v1:172.21.15.18:6790/0 conn(0x7fc5cc0b4700 legacy=0x7fc5cc0ab710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:31.776 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.775+0000 7fc5d364e640 1 -- 172.21.15.18:0/3420859074 shutdown_connections 2024-03-25T17:56:31.776 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.775+0000 7fc5d364e640 1 -- 172.21.15.18:0/3420859074 >> 172.21.15.18:0/3420859074 conn(0x7fc5cc0668e0 msgr2=0x7fc5cc0b6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:31.776 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.775+0000 7fc5d364e640 1 -- 172.21.15.18:0/3420859074 shutdown_connections 2024-03-25T17:56:31.777 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.775+0000 7fc5d364e640 1 -- 172.21.15.18:0/3420859074 wait complete. 2024-03-25T17:56:31.777 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-25T17:56:31.775+0000 7fc5d364e640 1 librados: shutdown 2024-03-25T17:56:31.791 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T17:56:31.792 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T17:56:32.598 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:56:32.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.744+0000 7f528c98f640 10 monclient: get_monmap_and_config 2024-03-25T17:56:32.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.744+0000 7f528c98f640 10 monclient: build_initial_monmap 2024-03-25T17:56:32.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.745+0000 7f528c98f640 10 monclient: monmap: 2024-03-25T17:56:32.746 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:56:32.746 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:32.746 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:56:32.746380+0000 2024-03-25T17:56:32.746 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:56:32.746380+0000 2024-03-25T17:56:32.746 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:32.746 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:32.746 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:32.746 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:56:32.746 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:56:32.747 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:32.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.746+0000 7f528c98f640 1 Processor -- start 2024-03-25T17:56:32.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.746+0000 7f528c98f640 1 -- start start 2024-03-25T17:56:32.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.746+0000 7f528c98f640 10 monclient: init 2024-03-25T17:56:32.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.747+0000 7f528c98f640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:32.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.747+0000 7f528c98f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:56:32.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.747+0000 7f528c98f640 10 monclient(hunting): picked mon.noname-b con 0x7f5280006800 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:32.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.747+0000 7f528c98f640 10 monclient(hunting): picked mon.noname-c con 0x7f5280006be0 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:32.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.747+0000 7f528c98f640 10 monclient(hunting): picked mon.noname-a con 0x7f52800c95e0 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:32.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.747+0000 7f528c98f640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52800c5570 con 0x7f52800c95e0 2024-03-25T17:56:32.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.747+0000 7f528c98f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528002e450 con 0x7f5280006be0 2024-03-25T17:56:32.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.747+0000 7f528c98f640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52800c31c0 con 0x7f5280006800 2024-03-25T17:56:32.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.747+0000 7f528c98f640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:32.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.747+0000 7f528c98f640 10 monclient(hunting): authenticate will time out at 6921.325797s 2024-03-25T17:56:32.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.748+0000 7f5286d9d640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f5280006800 0x7f52800c2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:54274/0 (socket says 172.21.15.18:54274) 2024-03-25T17:56:32.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.748+0000 7f5286d9d640 1 -- 172.21.15.18:0/1246351480 learned_addr learned my addr 172.21.15.18:0/1246351480 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:56:32.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.780+0000 7f5285d9b640 1 -- 172.21.15.18:0/1246351480 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298599260 0 0) 0x7f52800c5570 con 0x7f52800c95e0 2024-03-25T17:56:32.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.781+0000 7f5285d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:56:32.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.781+0000 7f5285d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:32.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.781+0000 7f5285d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:32.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.781+0000 7f5285d9b640 10 monclient(hunting): my global_id is 4176 2024-03-25T17:56:32.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.781+0000 7f5285d9b640 1 -- 172.21.15.18:0/1246351480 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5270003520 con 0x7f52800c95e0 2024-03-25T17:56:32.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.785+0000 7f5285d9b640 1 -- 172.21.15.18:0/1246351480 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605665657 0 0) 0x7f52800c31c0 con 0x7f5280006800 2024-03-25T17:56:32.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.785+0000 7f5285d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:56:32.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.785+0000 7f5285d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:32.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.785+0000 7f5285d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:32.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.785+0000 7f5285d9b640 10 monclient(hunting): my global_id is 4184 2024-03-25T17:56:32.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.785+0000 7f5285d9b640 1 -- 172.21.15.18:0/1246351480 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52800c5570 con 0x7f5280006800 2024-03-25T17:56:32.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.791+0000 7f5285d9b640 1 -- 172.21.15.18:0/1246351480 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2222401927 0 0) 0x7f52800c5570 con 0x7f5280006800 2024-03-25T17:56:32.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.791+0000 7f5285d9b640 1 -- 172.21.15.18:0/1246351480 >> v1:172.21.15.169:6789/0 conn(0x7f5280006be0 legacy=0x7f52800c8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:56:32.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.791+0000 7f5285d9b640 1 -- 172.21.15.18:0/1246351480 >> v1:172.21.15.18:6789/0 conn(0x7f52800c95e0 legacy=0x7f52800d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:32.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.791+0000 7f5285d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:56:32.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.791+0000 7f5285d9b640 1 monclient: found mon.noname-b 2024-03-25T17:56:32.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.791+0000 7f5285d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:32.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.791+0000 7f5285d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T17:56:32.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.791+0000 7f5285d9b640 1 -- 172.21.15.18:0/1246351480 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528002cde0 con 0x7f5280006800 2024-03-25T17:56:32.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.791+0000 7f5285d9b640 10 monclient: _finish_auth 0 2024-03-25T17:56:32.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.791+0000 7f5285d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:56:32.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.791+0000 7f5285d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:32.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.791+0000 7f5285d9b640 1 -- 172.21.15.18:0/1246351480 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5278001690 con 0x7f5280006800 2024-03-25T17:56:32.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.791+0000 7f5285d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:32.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.791+0000 7f5285d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:56:32.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.791+0000 7f5285d9b640 10 monclient: dump: 2024-03-25T17:56:32.793 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:32.793 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:32.793 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:32.793 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:32.793 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:32.793 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:32.793 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:32.793 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:32.793 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:32.793 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:32.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.791+0000 7f528c98f640 5 monclient: authenticate success, global_id 4184 2024-03-25T17:56:32.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.791+0000 7f528c98f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:32.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.799+0000 7f5285d9b640 1 -- 172.21.15.18:0/1246351480 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52780017f0 con 0x7f5280006800 2024-03-25T17:56:32.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.799+0000 7f5285d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:32.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.799+0000 7f5285d9b640 1 -- 172.21.15.18:0/1246351480 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5278001c00 con 0x7f5280006800 2024-03-25T17:56:32.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.799+0000 7f5285d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:32.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.799+0000 7f5285d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:56:32.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.799+0000 7f5285d9b640 10 monclient: dump: 2024-03-25T17:56:32.800 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:32.800 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:32.800 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:32.800 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:32.800 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:32.800 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:32.800 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:32.800 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:32.800 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:32.800 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:32.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.799+0000 7f528c98f640 10 monclient: get_monmap_and_config success 2024-03-25T17:56:32.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.799+0000 7f528c98f640 4 set_mon_vals no callback set 2024-03-25T17:56:32.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.800+0000 7f528c98f640 10 monclient: shutdown 2024-03-25T17:56:32.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.800+0000 7f528c98f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:32.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.800+0000 7f528c98f640 1 -- 172.21.15.18:0/1246351480 >> v1:172.21.15.18:6790/0 conn(0x7f5280006800 legacy=0x7f52800c2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:32.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.800+0000 7f528c98f640 1 -- 172.21.15.18:0/1246351480 shutdown_connections 2024-03-25T17:56:32.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.800+0000 7f528c98f640 1 -- 172.21.15.18:0/1246351480 >> 172.21.15.18:0/1246351480 conn(0x7f52800058f0 msgr2=0x7f5280005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:32.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.800+0000 7f528c98f640 1 -- 172.21.15.18:0/1246351480 shutdown_connections 2024-03-25T17:56:32.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.800+0000 7f528c98f640 1 -- 172.21.15.18:0/1246351480 wait complete. 2024-03-25T17:56:32.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.801+0000 7f528c98f640 10 monclient: build_initial_monmap 2024-03-25T17:56:32.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.801+0000 7f528c98f640 10 monclient: monmap: 2024-03-25T17:56:32.803 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:56:32.803 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:56:32.803 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:56:32.803 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:56:32.803 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:32.803 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:56:32.803 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:32.803 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:56:32.804 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:56:32.804 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:32.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.802+0000 7f528c98f640 1 librados: starting msgr at 2024-03-25T17:56:32.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.802+0000 7f528c98f640 1 librados: starting objecter 2024-03-25T17:56:32.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.802+0000 7f528c98f640 1 Processor -- start 2024-03-25T17:56:32.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.802+0000 7f528c98f640 1 -- start start 2024-03-25T17:56:32.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.802+0000 7f528c98f640 1 librados: setting wanted keys 2024-03-25T17:56:32.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.802+0000 7f528c98f640 1 librados: calling monclient init 2024-03-25T17:56:32.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.802+0000 7f528c98f640 10 monclient: init 2024-03-25T17:56:32.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.803+0000 7f528c98f640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:32.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.803+0000 7f528c98f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:56:32.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.803+0000 7f528c98f640 10 monclient(hunting): picked mon.noname-a con 0x7f5280006800 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:32.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.803+0000 7f528c98f640 10 monclient(hunting): picked mon.noname-b con 0x7f52800c95e0 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:32.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.803+0000 7f528c98f640 10 monclient(hunting): picked mon.noname-c con 0x7f52800c4920 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:32.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.803+0000 7f528c98f640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528002ff90 con 0x7f5280006800 2024-03-25T17:56:32.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.803+0000 7f528c98f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5280030200 con 0x7f52800c95e0 2024-03-25T17:56:32.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.804+0000 7f528759e640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f52800c4920 0x7f528016b4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:54290/0 (socket says 172.21.15.18:54290) 2024-03-25T17:56:32.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.804+0000 7f528759e640 1 -- 172.21.15.18:0/105792222 learned_addr learned my addr 172.21.15.18:0/105792222 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:56:32.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.804+0000 7f528c98f640 1 -- 172.21.15.18:0/105792222 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52800c56f0 con 0x7f52800c4920 2024-03-25T17:56:32.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.804+0000 7f528c98f640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:32.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.804+0000 7f526f7fe640 10 client.?.objecter ms_handle_connect 0x7f52800c4920 2024-03-25T17:56:32.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.804+0000 7f526f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:32.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.804+0000 7f526f7fe640 10 client.?.objecter ms_handle_connect 0x7f5280006800 2024-03-25T17:56:32.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.804+0000 7f526f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:32.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.804+0000 7f528c98f640 10 monclient(hunting): authenticate will time out at 6921.383319s 2024-03-25T17:56:32.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.808+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792483801 0 0) 0x7f52800c56f0 con 0x7f52800c4920 2024-03-25T17:56:32.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.808+0000 7f526f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:56:32.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.808+0000 7f526f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:32.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.808+0000 7f526f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:32.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.808+0000 7f526f7fe640 10 monclient(hunting): my global_id is 4187 2024-03-25T17:56:32.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.808+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5274004eb0 con 0x7f52800c4920 2024-03-25T17:56:32.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.809+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181437301 0 0) 0x7f528002ff90 con 0x7f5280006800 2024-03-25T17:56:32.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.809+0000 7f526f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:56:32.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.809+0000 7f526f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:32.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.809+0000 7f526f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:32.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.809+0000 7f526f7fe640 10 monclient(hunting): my global_id is 4179 2024-03-25T17:56:32.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.809+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52800c56f0 con 0x7f5280006800 2024-03-25T17:56:32.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.816+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2156555229 0 0) 0x7f5274004eb0 con 0x7f52800c4920 2024-03-25T17:56:32.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.816+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f528002ff90 con 0x7f52800c4920 2024-03-25T17:56:32.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.816+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5288069d20 con 0x7f52800c4920 2024-03-25T17:56:32.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.816+0000 7f526f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:32.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.816+0000 7f526f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:56:32.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.816+0000 7f526f7fe640 10 monclient(hunting): dump: 2024-03-25T17:56:32.817 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:32.817 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:32.817 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:32.817 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:32.817 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:32.817 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:32.817 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:32.817 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:32.817 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:32.817 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:32.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.816+0000 7f526f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:56:32.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.818+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2541622896 0 0) 0x7f52800c56f0 con 0x7f5280006800 2024-03-25T17:56:32.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.818+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5274004eb0 con 0x7f5280006800 2024-03-25T17:56:32.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.819+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5278001ba0 con 0x7f5280006800 2024-03-25T17:56:32.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.819+0000 7f526f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:32.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.819+0000 7f526f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:32.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.819+0000 7f526f7fe640 10 monclient(hunting): dump: 2024-03-25T17:56:32.820 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:32.820 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:32.820 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:32.820 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:32.820 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:32.820 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:32.820 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:32.820 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:32.820 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:32.821 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:32.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.822+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4003668756 0 0) 0x7f528002ff90 con 0x7f52800c4920 2024-03-25T17:56:32.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.822+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 >> v1:172.21.15.169:6789/0 conn(0x7f52800c95e0 legacy=0x7f5280164920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:32.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.822+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 >> v1:172.21.15.18:6789/0 conn(0x7f5280006800 legacy=0x7f5280164210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:32.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.822+0000 7f526f7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:56:32.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.822+0000 7f526f7fe640 1 monclient: found mon.c 2024-03-25T17:56:32.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.822+0000 7f526f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:32.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.822+0000 7f526f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:56:32.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.822+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52800c1980 con 0x7f52800c4920 2024-03-25T17:56:32.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.822+0000 7f526f7fe640 10 monclient: _finish_auth 0 2024-03-25T17:56:32.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.822+0000 7f526f7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:56:32.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.822+0000 7f526f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:32.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.822+0000 7f528c98f640 5 monclient: authenticate success, global_id 4187 2024-03-25T17:56:32.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.822+0000 7f528c98f640 10 monclient: _renew_subs 2024-03-25T17:56:32.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.822+0000 7f528c98f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:56:32.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.822+0000 7f528c98f640 1 -- 172.21.15.18:0/105792222 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52800c53c0 con 0x7f52800c4920 2024-03-25T17:56:32.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.823+0000 7f528c98f640 10 client.4187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:56:32.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.823+0000 7f528c98f640 10 monclient: _renew_subs 2024-03-25T17:56:32.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.823+0000 7f528c98f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:56:32.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.823+0000 7f528c98f640 1 -- 172.21.15.18:0/105792222 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f528016cc00 con 0x7f52800c4920 2024-03-25T17:56:32.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.823+0000 7f528c98f640 1 librados: init done 2024-03-25T17:56:32.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.824+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5288075d50 con 0x7f52800c4920 2024-03-25T17:56:32.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.824+0000 7f526f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:32.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.824+0000 7f528c98f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:32.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.825+0000 7f528559a640 4 set_mon_vals no callback set 2024-03-25T17:56:32.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.825+0000 7f528c98f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:32.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.825+0000 7f528c98f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:56:32.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.825+0000 7f528c98f640 1 -- 172.21.15.18:0/105792222 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5280004cd0 con 0x7f52800c4920 2024-03-25T17:56:32.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.827+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f528804f450 con 0x7f52800c4920 2024-03-25T17:56:32.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.827+0000 7f526f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:32.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.827+0000 7f526f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:56:32.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.827+0000 7f526f7fe640 10 monclient: dump: 2024-03-25T17:56:32.830 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:32.831 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:32.831 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:32.831 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:32.831 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:32.831 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:32.831 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:32.831 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:32.831 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:32.831 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:32.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.893+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 3) v1 ==== 89499+0+0 (unknown 1278801531 0 0) 0x7f5288067c80 con 0x7f52800c4920 2024-03-25T17:56:32.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.897+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f52880cf780 con 0x7f52800c4920 2024-03-25T17:56:32.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.897+0000 7f526f7fe640 10 client.4187.objecter ms_dispatch 0x7f5288000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:56:32.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.897+0000 7f526f7fe640 3 client.4187.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:56:32.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.897+0000 7f526f7fe640 3 client.4187.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:56:32.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:32.898+0000 7f526f7fe640 20 client.4187.objecter dump_active .. 0 homeless 2024-03-25T17:56:33.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.037+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f52880cb0c0 con 0x7f52800c4920 2024-03-25T17:56:33.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.037+0000 7f526f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:33.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.037+0000 7f526f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:56:33.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.167+0000 7f528c98f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:56:33.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.167+0000 7f528c98f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:56:33.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.167+0000 7f528c98f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:56:33.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.167+0000 7f528c98f640 1 -- 172.21.15.18:0/105792222 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52800c5550 con 0x7f52800c4920 2024-03-25T17:56:33.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.198+0000 7f526f7fe640 1 -- 172.21.15.18:0/105792222 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f52800c5550 con 0x7f52800c4920 2024-03-25T17:56:33.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.198+0000 7f526f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:56:33.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.198+0000 7f526f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:56:33.199 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:56:33.199 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:56:33.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.199+0000 7f528c98f640 10 librados: watch_flush enter 2024-03-25T17:56:33.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.199+0000 7f528c98f640 10 librados: watch_flush exit 2024-03-25T17:56:33.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.199+0000 7f528c98f640 20 client.4187.objecter shutdown clearing up homeless session... 2024-03-25T17:56:33.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.199+0000 7f528c98f640 10 client.4187.objecter successfully canceled tick 2024-03-25T17:56:33.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.199+0000 7f528c98f640 10 monclient: shutdown 2024-03-25T17:56:33.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.199+0000 7f528c98f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:33.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.199+0000 7f528c98f640 1 -- 172.21.15.18:0/105792222 >> v1:172.21.15.18:6790/0 conn(0x7f52800c4920 legacy=0x7f528016b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:33.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.199+0000 7f528c98f640 1 -- 172.21.15.18:0/105792222 shutdown_connections 2024-03-25T17:56:33.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.199+0000 7f528c98f640 1 -- 172.21.15.18:0/105792222 >> 172.21.15.18:0/105792222 conn(0x7f52800058f0 msgr2=0x7f52800050f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:33.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.199+0000 7f528c98f640 1 -- 172.21.15.18:0/105792222 shutdown_connections 2024-03-25T17:56:33.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.199+0000 7f528c98f640 1 -- 172.21.15.18:0/105792222 wait complete. 2024-03-25T17:56:33.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:33.199+0000 7f528c98f640 1 librados: shutdown 2024-03-25T17:56:33.214 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:56:34.172 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:34.171+0000 7fc3f02c2640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T17:56:34.211 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-25T17:56:34.210+0000 7fc3f02c2640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T17:56:37.976 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:06.725 35734== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:38.036 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:06.785 35734== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:38.071 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:06.811 35736== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:38.122 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:06.881 35733== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:38.135 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:06.876 35736== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:38.182 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:06.942 35733== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:38.449 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:07.194 35735== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:38.508 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:07.252 35735== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:38.593 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:07.345 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:38.599 INFO:tasks.ceph.osd.5.smithi169.stderr:2024-03-25T17:56:38.592+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:56:38.658 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:07.411 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:38.684 INFO:tasks.ceph.osd.7.smithi169.stderr:2024-03-25T17:56:38.677+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:56:38.729 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-03-25T17:56:38.722+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:56:38.748 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:07.485 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:38.814 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:07.550 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:38.883 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:07.625 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:38.944 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:07.686 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:39.052 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:07.802 35734== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:39.088 INFO:tasks.ceph.osd.4.smithi169.stderr:2024-03-25T17:56:39.079+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:56:39.168 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:07.908 35736== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:39.168 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:07.918 35734== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:39.190 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:07.949 35733== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:39.211 INFO:tasks.ceph.osd.1.smithi018.stderr:2024-03-25T17:56:39.204+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:56:39.277 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:08.026 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:39.339 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:08.087 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:39.366 INFO:tasks.ceph.osd.3.smithi018.stderr:2024-03-25T17:56:39.360+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:56:39.490 INFO:tasks.ceph.osd.2.smithi018.stderr:2024-03-25T17:56:39.485+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:56:39.534 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:08.274 35736== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:39.534 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:08.284 35734== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:39.563 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:08.322 35733== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:39.567 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:08.312 35735== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:39.668 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:08.420 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:39.829 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:08.565 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:39.850 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:08.600 35734== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:39.888 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:08.647 35733== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:39.888 INFO:tasks.ceph.osd.0.smithi018.stderr:2024-03-25T17:56:39.883+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:56:39.895 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:08.636 35736== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:39.953 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:08.698 35735== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:39.955 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:08.698 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.020 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:56:40.034 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:08.787 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.079 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:08.821 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.110+0000 7f40e8fb5640 10 monclient: get_monmap_and_config 2024-03-25T17:56:40.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.110+0000 7f40e8fb5640 10 monclient: build_initial_monmap 2024-03-25T17:56:40.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.110+0000 7f40e8fb5640 10 monclient: monmap: 2024-03-25T17:56:40.111 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:56:40.111 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:40.111 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:56:40.111308+0000 2024-03-25T17:56:40.111 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:56:40.111308+0000 2024-03-25T17:56:40.111 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:40.111 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:40.111 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:40.111 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:56:40.111 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:56:40.111 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:40.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.111+0000 7f40e8fb5640 1 Processor -- start 2024-03-25T17:56:40.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.111+0000 7f40e8fb5640 1 -- start start 2024-03-25T17:56:40.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.111+0000 7f40e8fb5640 10 monclient: init 2024-03-25T17:56:40.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.112+0000 7f40e8fb5640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:40.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.112+0000 7f40e8fb5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:56:40.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.112+0000 7f40e8fb5640 10 monclient(hunting): picked mon.noname-b con 0x7f40e40b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:40.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.112+0000 7f40e8fb5640 10 monclient(hunting): picked mon.noname-c con 0x7f40e40b1730 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:40.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.112+0000 7f40e8fb5640 10 monclient(hunting): picked mon.noname-a con 0x7f40e414be90 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:40.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.112+0000 7f40e8fb5640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e40af940 con 0x7f40e414be90 2024-03-25T17:56:40.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.112+0000 7f40e8fb5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e40b0a70 con 0x7f40e40b1730 2024-03-25T17:56:40.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.112+0000 7f40e8fb5640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e40b0d50 con 0x7f40e40b4d60 2024-03-25T17:56:40.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.112+0000 7f40e8fb5640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:40.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.112+0000 7f40e8fb5640 10 monclient(hunting): authenticate will time out at 6928.690005s 2024-03-25T17:56:40.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.112+0000 7f40e259c640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f40e40b4d60 0x7f40e413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:40506/0 (socket says 172.21.15.18:40506) 2024-03-25T17:56:40.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.112+0000 7f40e259c640 1 -- 172.21.15.18:0/2172348956 learned_addr learned my addr 172.21.15.18:0/2172348956 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:56:40.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 1 -- 172.21.15.18:0/2172348956 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511668475 0 0) 0x7f40e40b0a70 con 0x7f40e40b1730 2024-03-25T17:56:40.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 10 monclient(hunting): _negotiate 2024-03-25T17:56:40.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:40.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:40.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 10 monclient(hunting): my global_id is 4201 2024-03-25T17:56:40.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 1 -- 172.21.15.18:0/2172348956 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40cc003520 con 0x7f40e40b1730 2024-03-25T17:56:40.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 1 -- 172.21.15.18:0/2172348956 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087461916 0 0) 0x7f40e40b0d50 con 0x7f40e40b4d60 2024-03-25T17:56:40.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 10 monclient(hunting): _negotiate 2024-03-25T17:56:40.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:40.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:40.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 10 monclient(hunting): my global_id is 4214 2024-03-25T17:56:40.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 1 -- 172.21.15.18:0/2172348956 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e40b0a70 con 0x7f40e40b4d60 2024-03-25T17:56:40.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 1 -- 172.21.15.18:0/2172348956 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402330504 0 0) 0x7f40e40af940 con 0x7f40e414be90 2024-03-25T17:56:40.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 10 monclient(hunting): _negotiate 2024-03-25T17:56:40.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:40.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:40.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 10 monclient(hunting): my global_id is 4206 2024-03-25T17:56:40.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.115+0000 7f40e159a640 1 -- 172.21.15.18:0/2172348956 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e40b0d50 con 0x7f40e414be90 2024-03-25T17:56:40.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.118+0000 7f40e159a640 1 -- 172.21.15.18:0/2172348956 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4283646392 0 0) 0x7f40cc003520 con 0x7f40e40b1730 2024-03-25T17:56:40.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.118+0000 7f40e159a640 1 -- 172.21.15.18:0/2172348956 >> v1:172.21.15.18:6790/0 conn(0x7f40e40b4d60 legacy=0x7f40e413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:40.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.118+0000 7f40e159a640 1 -- 172.21.15.18:0/2172348956 >> v1:172.21.15.18:6789/0 conn(0x7f40e414be90 legacy=0x7f40e414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:40.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.118+0000 7f40e159a640 10 monclient: _finish_hunting 0 2024-03-25T17:56:40.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.118+0000 7f40e159a640 1 monclient: found mon.noname-c 2024-03-25T17:56:40.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.118+0000 7f40e159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:40.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.118+0000 7f40e159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T17:56:40.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.118+0000 7f40e159a640 1 -- 172.21.15.18:0/2172348956 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e40b1030 con 0x7f40e40b1730 2024-03-25T17:56:40.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.118+0000 7f40e159a640 10 monclient: _finish_auth 0 2024-03-25T17:56:40.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.118+0000 7f40e159a640 10 monclient: _check_auth_tickets 2024-03-25T17:56:40.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.118+0000 7f40e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:40.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.118+0000 7f40e159a640 1 -- 172.21.15.18:0/2172348956 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f40d8001690 con 0x7f40e40b1730 2024-03-25T17:56:40.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.118+0000 7f40e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:40.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.118+0000 7f40e159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:56:40.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.118+0000 7f40e159a640 10 monclient: dump: 2024-03-25T17:56:40.119 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:40.119 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:40.119 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:40.119 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:40.120 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:40.120 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:40.120 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:40.120 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:40.120 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:40.120 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:40.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.118+0000 7f40e8fb5640 5 monclient: authenticate success, global_id 4201 2024-03-25T17:56:40.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.118+0000 7f40e8fb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:40.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.120+0000 7f40e159a640 1 -- 172.21.15.18:0/2172348956 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40d80017f0 con 0x7f40e40b1730 2024-03-25T17:56:40.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.120+0000 7f40e159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:40.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.120+0000 7f40e159a640 1 -- 172.21.15.18:0/2172348956 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f40d8001c00 con 0x7f40e40b1730 2024-03-25T17:56:40.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.120+0000 7f40e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:40.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.120+0000 7f40e159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:56:40.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.120+0000 7f40e159a640 10 monclient: dump: 2024-03-25T17:56:40.123 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:40.123 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:40.123 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:40.123 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:40.123 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:40.123 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:40.123 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:40.123 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:40.123 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:40.123 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:40.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.120+0000 7f40e8fb5640 10 monclient: get_monmap_and_config success 2024-03-25T17:56:40.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.120+0000 7f40e8fb5640 4 set_mon_vals no callback set 2024-03-25T17:56:40.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.120+0000 7f40e8fb5640 10 monclient: shutdown 2024-03-25T17:56:40.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.120+0000 7f40e8fb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:40.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.120+0000 7f40e8fb5640 1 -- 172.21.15.18:0/2172348956 >> v1:172.21.15.169:6789/0 conn(0x7f40e40b1730 legacy=0x7f40e40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:40.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.121+0000 7f40e8fb5640 1 -- 172.21.15.18:0/2172348956 shutdown_connections 2024-03-25T17:56:40.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.121+0000 7f40e8fb5640 1 -- 172.21.15.18:0/2172348956 >> 172.21.15.18:0/2172348956 conn(0x7f40e40617a0 msgr2=0x7f40e4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:40.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.121+0000 7f40e8fb5640 1 -- 172.21.15.18:0/2172348956 shutdown_connections 2024-03-25T17:56:40.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.121+0000 7f40e8fb5640 1 -- 172.21.15.18:0/2172348956 wait complete. 2024-03-25T17:56:40.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.121+0000 7f40e8fb5640 10 monclient: build_initial_monmap 2024-03-25T17:56:40.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.121+0000 7f40e8fb5640 10 monclient: monmap: 2024-03-25T17:56:40.124 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:56:40.124 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:56:40.124 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:56:40.125 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:56:40.125 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:40.125 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:56:40.125 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:40.125 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:56:40.125 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:56:40.125 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:40.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.122+0000 7f40e8fb5640 1 librados: starting msgr at 2024-03-25T17:56:40.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.122+0000 7f40e8fb5640 1 librados: starting objecter 2024-03-25T17:56:40.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.122+0000 7f40e8fb5640 1 Processor -- start 2024-03-25T17:56:40.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.122+0000 7f40e8fb5640 1 -- start start 2024-03-25T17:56:40.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.122+0000 7f40e8fb5640 1 librados: setting wanted keys 2024-03-25T17:56:40.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.122+0000 7f40e8fb5640 1 librados: calling monclient init 2024-03-25T17:56:40.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.122+0000 7f40e8fb5640 10 monclient: init 2024-03-25T17:56:40.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.122+0000 7f40e8fb5640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:40.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.122+0000 7f40e8fb5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:56:40.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.122+0000 7f40e8fb5640 10 monclient(hunting): picked mon.noname-a con 0x7f40e40b1730 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:40.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.122+0000 7f40e8fb5640 10 monclient(hunting): picked mon.noname-c con 0x7f40e40b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:40.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.122+0000 7f40e8fb5640 10 monclient(hunting): picked mon.noname-b con 0x7f40e414be90 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:40.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.122+0000 7f40e8fb5640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e413bfb0 con 0x7f40e40b1730 2024-03-25T17:56:40.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.122+0000 7f40e8fb5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e41458e0 con 0x7f40e414be90 2024-03-25T17:56:40.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.122+0000 7f40e8fb5640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40e412f7b0 con 0x7f40e40b4d60 2024-03-25T17:56:40.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.122+0000 7f40e8fb5640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:40.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.122+0000 7f40e8fb5640 10 monclient(hunting): authenticate will time out at 6928.700611s 2024-03-25T17:56:40.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.123+0000 7f40e259c640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f40e40b1730 0x7f40e413b8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:58992/0 (socket says 172.21.15.18:58992) 2024-03-25T17:56:40.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.123+0000 7f40e259c640 1 -- 172.21.15.18:0/4118434428 learned_addr learned my addr 172.21.15.18:0/4118434428 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:56:40.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.123+0000 7f40d2ffd640 10 client.?.objecter ms_handle_connect 0x7f40e40b1730 2024-03-25T17:56:40.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.123+0000 7f40d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:40.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.123+0000 7f40d2ffd640 10 client.?.objecter ms_handle_connect 0x7f40e40b4d60 2024-03-25T17:56:40.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.123+0000 7f40d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:40.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.124+0000 7f40d2ffd640 10 client.?.objecter ms_handle_connect 0x7f40e414be90 2024-03-25T17:56:40.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.124+0000 7f40d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:40.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.125+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140234472 0 0) 0x7f40e41458e0 con 0x7f40e414be90 2024-03-25T17:56:40.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.125+0000 7f40d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:56:40.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.125+0000 7f40d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:40.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.125+0000 7f40d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:40.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.125+0000 7f40d2ffd640 10 monclient(hunting): my global_id is 4204 2024-03-25T17:56:40.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.125+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40c4004eb0 con 0x7f40e414be90 2024-03-25T17:56:40.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.126+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559322750 0 0) 0x7f40e413bfb0 con 0x7f40e40b1730 2024-03-25T17:56:40.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.126+0000 7f40d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:56:40.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.126+0000 7f40d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:40.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.126+0000 7f40d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:40.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.126+0000 7f40d2ffd640 10 monclient(hunting): my global_id is 4209 2024-03-25T17:56:40.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.126+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e41458e0 con 0x7f40e40b1730 2024-03-25T17:56:40.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.127+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413855425 0 0) 0x7f40e412f7b0 con 0x7f40e40b4d60 2024-03-25T17:56:40.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.127+0000 7f40d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:56:40.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.127+0000 7f40d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:40.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.127+0000 7f40d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:40.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.127+0000 7f40d2ffd640 10 monclient(hunting): my global_id is 4217 2024-03-25T17:56:40.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.127+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e413bfb0 con 0x7f40e40b4d60 2024-03-25T17:56:40.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.130+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1159508287 0 0) 0x7f40c4004eb0 con 0x7f40e414be90 2024-03-25T17:56:40.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.130+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40e412f7b0 con 0x7f40e414be90 2024-03-25T17:56:40.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.130+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f40dc001bd0 con 0x7f40e414be90 2024-03-25T17:56:40.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.130+0000 7f40d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:40.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.130+0000 7f40d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:56:40.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.130+0000 7f40d2ffd640 10 monclient(hunting): dump: 2024-03-25T17:56:40.131 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:40.131 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:40.131 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:40.132 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:40.132 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:40.132 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:40.132 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:40.132 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:40.132 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:40.132 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:40.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.130+0000 7f40d2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:56:40.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.131+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2796249775 0 0) 0x7f40e41458e0 con 0x7f40e40b1730 2024-03-25T17:56:40.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.131+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40c4004eb0 con 0x7f40e40b1730 2024-03-25T17:56:40.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.131+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f40d4001970 con 0x7f40e40b1730 2024-03-25T17:56:40.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.131+0000 7f40d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:40.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.131+0000 7f40d2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:40.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.131+0000 7f40d2ffd640 10 monclient(hunting): dump: 2024-03-25T17:56:40.133 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:40.133 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:40.133 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:40.133 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:40.133 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:40.133 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:40.133 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:40.133 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:40.133 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:40.133 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:40.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 620876805 0 0) 0x7f40e412f7b0 con 0x7f40e414be90 2024-03-25T17:56:40.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 >> v1:172.21.15.18:6790/0 conn(0x7f40e40b4d60 legacy=0x7f40e412e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:40.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 >> v1:172.21.15.18:6789/0 conn(0x7f40e40b1730 legacy=0x7f40e413b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:40.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:56:40.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40d2ffd640 1 monclient: found mon.b 2024-03-25T17:56:40.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:40.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40d2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:40.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e413aa90 con 0x7f40e414be90 2024-03-25T17:56:40.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40d2ffd640 10 monclient: _finish_auth 0 2024-03-25T17:56:40.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:56:40.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:40.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40e8fb5640 5 monclient: authenticate success, global_id 4204 2024-03-25T17:56:40.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40e8fb5640 10 monclient: _renew_subs 2024-03-25T17:56:40.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40e8fb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:40.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40e8fb5640 1 -- 172.21.15.18:0/4118434428 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e413ad70 con 0x7f40e414be90 2024-03-25T17:56:40.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40e8fb5640 10 client.4204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:56:40.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40e8fb5640 10 monclient: _renew_subs 2024-03-25T17:56:40.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40e8fb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:40.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40e8fb5640 1 -- 172.21.15.18:0/4118434428 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e4145330 con 0x7f40e414be90 2024-03-25T17:56:40.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.133+0000 7f40e8fb5640 1 librados: init done 2024-03-25T17:56:40.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.134+0000 7f40e8fb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:40.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.134+0000 7f40e8fb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:40.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.134+0000 7f40e8fb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:40.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.134+0000 7f40e8fb5640 1 -- 172.21.15.18:0/4118434428 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40e41401f0 con 0x7f40e414be90 2024-03-25T17:56:40.150 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:08.899 35734== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.165 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:08.924 35733== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.190 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:08.930 35736== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.203 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:08.939 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.308 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:09.053 35735== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.357 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:09.105 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.358 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:09.111 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.379+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40dc002060 con 0x7f40e414be90 2024-03-25T17:56:40.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.379+0000 7f40d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:40.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.379+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f40dc003b30 con 0x7f40e414be90 2024-03-25T17:56:40.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.379+0000 7f40d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:40.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.379+0000 7f40d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:56:40.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.379+0000 7f40d2ffd640 10 monclient: dump: 2024-03-25T17:56:40.379 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:40.379 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:40.379 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:40.380 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:40.380 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:40.380 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:40.380 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:40.380 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:40.380 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:40.380 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:40.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.379+0000 7f40e0d99640 4 set_mon_vals no callback set 2024-03-25T17:56:40.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.380+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f40dc019b00 con 0x7f40e414be90 2024-03-25T17:56:40.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.380+0000 7f40e259c640 10 monclient: get_auth_request con 0x7f40c406df20 auth_method 0 2024-03-25T17:56:40.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.380+0000 7f40d2ffd640 10 client.4204.objecter ms_handle_connect 0x7f40c406df20 2024-03-25T17:56:40.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.383+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f40dc0537a0 con 0x7f40e414be90 2024-03-25T17:56:40.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.383+0000 7f40d2ffd640 10 client.4204.objecter ms_dispatch 0x7f40e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:56:40.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.383+0000 7f40d2ffd640 3 client.4204.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:56:40.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.383+0000 7f40d2ffd640 3 client.4204.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:56:40.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.383+0000 7f40d2ffd640 20 client.4204.objecter dump_active .. 0 homeless 2024-03-25T17:56:40.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.383+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f40dc04f2b0 con 0x7f40e414be90 2024-03-25T17:56:40.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.383+0000 7f40d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:40.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.383+0000 7f40d2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:56:40.411 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:09.153 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.442 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:09.200 35733== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.450 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:09.200 35734== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.492 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:09.232 35736== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.560 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:09.296 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.568+0000 7f40e8fb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:56:40.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.568+0000 7f40e8fb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:56:40.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.568+0000 7f40e8fb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:40.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.568+0000 7f40e8fb5640 1 -- 172.21.15.18:0/4118434428 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40e40afbd0 con 0x7f40e414be90 2024-03-25T17:56:40.596 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:09.341 35735== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.617+0000 7f40d2ffd640 1 -- 172.21.15.18:0/4118434428 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f40dc0511e0 con 0x7f40e414be90 2024-03-25T17:56:40.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.617+0000 7f40d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:56:40.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.617+0000 7f40d2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:56:40.617 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:56:40.618 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:56:40.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.620+0000 7f40e8fb5640 10 librados: watch_flush enter 2024-03-25T17:56:40.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.620+0000 7f40e8fb5640 10 librados: watch_flush exit 2024-03-25T17:56:40.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.620+0000 7f40e8fb5640 20 client.4204.objecter shutdown clearing up homeless session... 2024-03-25T17:56:40.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.620+0000 7f40e8fb5640 10 client.4204.objecter successfully canceled tick 2024-03-25T17:56:40.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.620+0000 7f40e8fb5640 1 -- 172.21.15.18:0/4118434428 >> v1:172.21.15.18:6800/3721244451 conn(0x7f40c406df20 legacy=0x7f40c40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:40.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.620+0000 7f40e8fb5640 10 monclient: shutdown 2024-03-25T17:56:40.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.620+0000 7f40e8fb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:40.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.620+0000 7f40e8fb5640 1 -- 172.21.15.18:0/4118434428 >> v1:172.21.15.169:6789/0 conn(0x7f40e414be90 legacy=0x7f40e412f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:40.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.620+0000 7f40e8fb5640 1 -- 172.21.15.18:0/4118434428 shutdown_connections 2024-03-25T17:56:40.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.620+0000 7f40e8fb5640 1 -- 172.21.15.18:0/4118434428 >> 172.21.15.18:0/4118434428 conn(0x7f40e40617a0 msgr2=0x7f40e413c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:40.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.620+0000 7f40e8fb5640 1 -- 172.21.15.18:0/4118434428 shutdown_connections 2024-03-25T17:56:40.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.620+0000 7f40e8fb5640 1 -- 172.21.15.18:0/4118434428 wait complete. 2024-03-25T17:56:40.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:40.620+0000 7f40e8fb5640 1 librados: shutdown 2024-03-25T17:56:40.629 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:56:40.636 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:09.388 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.683 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:09.425 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.708 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:09.468 35733== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.738 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:09.488 35734== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.750 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:09.499 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.779 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:09.519 35736== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.848 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:09.585 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.897 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:09.642 35735== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.900 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:09.652 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.947 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:09.689 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:40.977 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:09.736 35733== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.027 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:09.776 35734== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.049 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:09.798 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.076 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:09.816 35736== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.137 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:09.874 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.166 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:09.918 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.185 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:09.930 35735== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.215 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:09.957 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.241 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:10.000 35733== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.314 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:10.063 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.318 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:10.067 35734== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.368 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:10.108 35736== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.431 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:10.167 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.434 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:10.186 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.476 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:10.221 35735== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.482 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:10.221 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.512 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:10.272 35733== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.580 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:10.329 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.658 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:10.398 35736== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.698 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:10.450 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.725 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:10.462 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.747 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:10.489 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.763 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:10.508 35735== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.824 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:10.574 35734== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.844 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:10.593 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.879 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:10.628 35734== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.967 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:10.719 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:41.989 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:10.748 35733== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.015 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:10.757 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.018 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:10.755 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.043 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:10.802 35733== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.069 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:10.814 35735== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.109 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:10.858 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.176 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:10.916 35736== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.230 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:10.970 35736== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.310 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:11.047 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.382 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:11.131 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.446 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:11.199 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.494 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:11.237 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.506 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:11.258 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.570 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:11.312 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.595 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:11.340 35735== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.602 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:11.339 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.658 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:11.407 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.658 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:11.395 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:42.668 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:11.413 35735== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:43.162 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:11.910 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:43.217 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:11.966 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:47.436 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:56:47.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.558+0000 7f2105c8d640 10 monclient: get_monmap_and_config 2024-03-25T17:56:47.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.558+0000 7f2105c8d640 10 monclient: build_initial_monmap 2024-03-25T17:56:47.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.558+0000 7f2105c8d640 10 monclient: monmap: 2024-03-25T17:56:47.558 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:56:47.558 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:47.559 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:56:47.558985+0000 2024-03-25T17:56:47.559 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:56:47.558985+0000 2024-03-25T17:56:47.559 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:47.559 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:47.559 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:47.559 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:56:47.559 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:56:47.559 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:47.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.559+0000 7f2105c8d640 1 Processor -- start 2024-03-25T17:56:47.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.559+0000 7f2105c8d640 1 -- start start 2024-03-25T17:56:47.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.559+0000 7f2105c8d640 10 monclient: init 2024-03-25T17:56:47.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.560+0000 7f2105c8d640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:47.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.560+0000 7f2105c8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:56:47.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.560+0000 7f2105c8d640 10 monclient(hunting): picked mon.noname-b con 0x7f21000a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:47.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.560+0000 7f2105c8d640 10 monclient(hunting): picked mon.noname-c con 0x7f210013f710 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:47.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.560+0000 7f2105c8d640 10 monclient(hunting): picked mon.noname-a con 0x7f21001430c0 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:47.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.560+0000 7f2105c8d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2100142520 con 0x7f21001430c0 2024-03-25T17:56:47.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.560+0000 7f2105c8d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21000b28a0 con 0x7f210013f710 2024-03-25T17:56:47.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.560+0000 7f2105c8d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21000b2a30 con 0x7f21000a98f0 2024-03-25T17:56:47.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.560+0000 7f2105c8d640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:47.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.560+0000 7f2105c8d640 10 monclient(hunting): authenticate will time out at 6936.137786s 2024-03-25T17:56:47.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.560+0000 7f20ff7fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f21000a98f0 0x7f21000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:55788/0 (socket says 172.21.15.18:55788) 2024-03-25T17:56:47.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.560+0000 7f20ff7fe640 1 -- 172.21.15.18:0/104733606 learned_addr learned my addr 172.21.15.18:0/104733606 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:56:47.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.563+0000 7f20fe7fc640 1 -- 172.21.15.18:0/104733606 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512049974 0 0) 0x7f2100142520 con 0x7f21001430c0 2024-03-25T17:56:47.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.563+0000 7f20fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:56:47.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.563+0000 7f20fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:47.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.563+0000 7f20fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:47.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.563+0000 7f20fe7fc640 10 monclient(hunting): my global_id is 4212 2024-03-25T17:56:47.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.563+0000 7f20fe7fc640 1 -- 172.21.15.18:0/104733606 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e8003520 con 0x7f21001430c0 2024-03-25T17:56:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.564+0000 7f20fe7fc640 1 -- 172.21.15.18:0/104733606 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3045960828 0 0) 0x7f21000b28a0 con 0x7f210013f710 2024-03-25T17:56:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.564+0000 7f20fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:56:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.564+0000 7f20fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.564+0000 7f20fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.564+0000 7f20fe7fc640 10 monclient(hunting): my global_id is 4207 2024-03-25T17:56:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.564+0000 7f20fe7fc640 1 -- 172.21.15.18:0/104733606 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2100142520 con 0x7f210013f710 2024-03-25T17:56:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.564+0000 7f20fe7fc640 1 -- 172.21.15.18:0/104733606 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712511701 0 0) 0x7f21000b2a30 con 0x7f21000a98f0 2024-03-25T17:56:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.564+0000 7f20fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:56:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.564+0000 7f20fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.564+0000 7f20fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:47.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.564+0000 7f20fe7fc640 10 monclient(hunting): my global_id is 4220 2024-03-25T17:56:47.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.564+0000 7f20fe7fc640 1 -- 172.21.15.18:0/104733606 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21000b28a0 con 0x7f21000a98f0 2024-03-25T17:56:47.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.566+0000 7f20fe7fc640 1 -- 172.21.15.18:0/104733606 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2616238893 0 0) 0x7f20e8003520 con 0x7f21001430c0 2024-03-25T17:56:47.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.566+0000 7f20fe7fc640 1 -- 172.21.15.18:0/104733606 >> v1:172.21.15.18:6790/0 conn(0x7f21000a98f0 legacy=0x7f21000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:47.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.567+0000 7f20fe7fc640 1 -- 172.21.15.18:0/104733606 >> v1:172.21.15.169:6789/0 conn(0x7f210013f710 legacy=0x7f21001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:47.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.567+0000 7f20fe7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:56:47.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.567+0000 7f20fe7fc640 1 monclient: found mon.noname-a 2024-03-25T17:56:47.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.567+0000 7f20fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:47.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.567+0000 7f20fe7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T17:56:47.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.567+0000 7f20fe7fc640 1 -- 172.21.15.18:0/104733606 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21000b44c0 con 0x7f21001430c0 2024-03-25T17:56:47.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.567+0000 7f20fe7fc640 10 monclient: _finish_auth 0 2024-03-25T17:56:47.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.567+0000 7f20fe7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:56:47.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.567+0000 7f20fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:47.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.567+0000 7f20fe7fc640 1 -- 172.21.15.18:0/104733606 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f20f8001a10 con 0x7f21001430c0 2024-03-25T17:56:47.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.567+0000 7f20fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:47.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.567+0000 7f20fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:56:47.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.567+0000 7f20fe7fc640 10 monclient: dump: 2024-03-25T17:56:47.568 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:47.568 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:47.568 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:47.568 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:47.568 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:47.568 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:47.568 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:47.568 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:47.568 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:47.568 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:47.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.567+0000 7f2105c8d640 5 monclient: authenticate success, global_id 4212 2024-03-25T17:56:47.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.567+0000 7f2105c8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:47.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.569+0000 7f20fe7fc640 1 -- 172.21.15.18:0/104733606 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20f8001c30 con 0x7f21001430c0 2024-03-25T17:56:47.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.569+0000 7f20fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:47.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.569+0000 7f2105c8d640 10 monclient: get_monmap_and_config success 2024-03-25T17:56:47.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.569+0000 7f2105c8d640 4 set_mon_vals no callback set 2024-03-25T17:56:47.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.569+0000 7f20fe7fc640 1 -- 172.21.15.18:0/104733606 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f20f8001fc0 con 0x7f21001430c0 2024-03-25T17:56:47.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.569+0000 7f20fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:47.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.569+0000 7f20fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:47.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.569+0000 7f20fe7fc640 10 monclient: dump: 2024-03-25T17:56:47.571 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:47.571 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:47.571 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:47.571 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:47.571 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:47.571 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:47.571 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:47.571 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:47.571 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:47.571 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:47.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.570+0000 7f2105c8d640 10 monclient: shutdown 2024-03-25T17:56:47.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.570+0000 7f2105c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:47.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.570+0000 7f2105c8d640 1 -- 172.21.15.18:0/104733606 >> v1:172.21.15.18:6789/0 conn(0x7f21001430c0 legacy=0x7f210014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:47.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.570+0000 7f2105c8d640 1 -- 172.21.15.18:0/104733606 shutdown_connections 2024-03-25T17:56:47.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.570+0000 7f2105c8d640 1 -- 172.21.15.18:0/104733606 >> 172.21.15.18:0/104733606 conn(0x7f21000b7ea0 msgr2=0x7f21000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:47.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.570+0000 7f2105c8d640 1 -- 172.21.15.18:0/104733606 shutdown_connections 2024-03-25T17:56:47.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.570+0000 7f2105c8d640 1 -- 172.21.15.18:0/104733606 wait complete. 2024-03-25T17:56:47.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.571+0000 7f2105c8d640 10 monclient: build_initial_monmap 2024-03-25T17:56:47.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.571+0000 7f2105c8d640 10 monclient: monmap: 2024-03-25T17:56:47.575 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:56:47.575 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:56:47.575 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:56:47.575 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:56:47.575 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:47.576 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:56:47.576 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:47.576 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:56:47.576 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:56:47.576 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:47.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.572+0000 7f2105c8d640 1 librados: starting msgr at 2024-03-25T17:56:47.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.572+0000 7f2105c8d640 1 librados: starting objecter 2024-03-25T17:56:47.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.572+0000 7f2105c8d640 1 Processor -- start 2024-03-25T17:56:47.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.572+0000 7f2105c8d640 1 -- start start 2024-03-25T17:56:47.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.572+0000 7f2105c8d640 1 librados: setting wanted keys 2024-03-25T17:56:47.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.572+0000 7f2105c8d640 1 librados: calling monclient init 2024-03-25T17:56:47.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.572+0000 7f2105c8d640 10 monclient: init 2024-03-25T17:56:47.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.573+0000 7f2105c8d640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:47.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.573+0000 7f2105c8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:56:47.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.573+0000 7f2105c8d640 10 monclient(hunting): picked mon.noname-c con 0x7f21000a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:47.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.573+0000 7f2105c8d640 10 monclient(hunting): picked mon.noname-a con 0x7f210013f710 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:47.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.574+0000 7f2105c8d640 10 monclient(hunting): picked mon.noname-b con 0x7f21001430c0 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:47.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.574+0000 7f20ff7fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f21000a98f0 0x7f21001daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:55800/0 (socket says 172.21.15.18:55800) 2024-03-25T17:56:47.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.574+0000 7f20ff7fe640 1 -- 172.21.15.18:0/1133448644 learned_addr learned my addr 172.21.15.18:0/1133448644 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:56:47.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.574+0000 7f2105c8d640 1 -- 172.21.15.18:0/1133448644 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2100136e20 con 0x7f210013f710 2024-03-25T17:56:47.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.574+0000 7f2105c8d640 1 -- 172.21.15.18:0/1133448644 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21000b7940 con 0x7f21001430c0 2024-03-25T17:56:47.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.574+0000 7f20e7fff640 10 client.?.objecter ms_handle_connect 0x7f21000a98f0 2024-03-25T17:56:47.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.574+0000 7f20e7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:47.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.574+0000 7f2105c8d640 1 -- 172.21.15.18:0/1133448644 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21001351a0 con 0x7f21000a98f0 2024-03-25T17:56:47.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.574+0000 7f2105c8d640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:47.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.574+0000 7f2105c8d640 10 monclient(hunting): authenticate will time out at 6936.152429s 2024-03-25T17:56:47.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.575+0000 7f20e7fff640 10 client.?.objecter ms_handle_connect 0x7f210013f710 2024-03-25T17:56:47.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.575+0000 7f20e7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:47.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.575+0000 7f20e7fff640 10 client.?.objecter ms_handle_connect 0x7f21001430c0 2024-03-25T17:56:47.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.575+0000 7f20e7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.576+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609969398 0 0) 0x7f21001351a0 con 0x7f21000a98f0 2024-03-25T17:56:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.576+0000 7f20e7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:56:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.576+0000 7f20e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.576+0000 7f20e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.576+0000 7f20e7fff640 10 monclient(hunting): my global_id is 4223 2024-03-25T17:56:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.576+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20ec004eb0 con 0x7f21000a98f0 2024-03-25T17:56:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.577+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488400621 0 0) 0x7f21000b7940 con 0x7f21001430c0 2024-03-25T17:56:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.577+0000 7f20e7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:56:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.577+0000 7f20e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.577+0000 7f20e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.577+0000 7f20e7fff640 10 monclient(hunting): my global_id is 4210 2024-03-25T17:56:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.577+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21001351a0 con 0x7f21001430c0 2024-03-25T17:56:47.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.578+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296018668 0 0) 0x7f2100136e20 con 0x7f210013f710 2024-03-25T17:56:47.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.578+0000 7f20e7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:56:47.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.578+0000 7f20e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:47.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.578+0000 7f20e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:47.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.578+0000 7f20e7fff640 10 monclient(hunting): my global_id is 4215 2024-03-25T17:56:47.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.578+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21000b7940 con 0x7f210013f710 2024-03-25T17:56:47.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.583+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3992815836 0 0) 0x7f21000b7940 con 0x7f210013f710 2024-03-25T17:56:47.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.583+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2100136e20 con 0x7f210013f710 2024-03-25T17:56:47.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.583+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f20f40018f0 con 0x7f210013f710 2024-03-25T17:56:47.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.583+0000 7f20e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:47.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.583+0000 7f20e7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:56:47.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.583+0000 7f20e7fff640 10 monclient(hunting): dump: 2024-03-25T17:56:47.583 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:47.583 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:47.583 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:47.583 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:47.583 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:47.583 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:47.583 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:47.584 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:47.584 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:47.584 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:47.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.583+0000 7f20e7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:56:47.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.583+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3540010955 0 0) 0x7f20ec004eb0 con 0x7f21000a98f0 2024-03-25T17:56:47.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.583+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21000b7940 con 0x7f21000a98f0 2024-03-25T17:56:47.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.585+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f20f00019f0 con 0x7f21000a98f0 2024-03-25T17:56:47.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.585+0000 7f20e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:47.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.585+0000 7f20e7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:56:47.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.585+0000 7f20e7fff640 10 monclient(hunting): dump: 2024-03-25T17:56:47.585 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:47.585 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:47.586 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:47.586 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:47.586 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:47.586 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:47.586 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:47.586 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:47.586 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:47.586 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:47.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.585+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3645697884 0 0) 0x7f21001351a0 con 0x7f21001430c0 2024-03-25T17:56:47.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.585+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20ec004eb0 con 0x7f21001430c0 2024-03-25T17:56:47.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.585+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f20f8001da0 con 0x7f21001430c0 2024-03-25T17:56:47.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.585+0000 7f20e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:47.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.585+0000 7f20e7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:56:47.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.585+0000 7f20e7fff640 10 monclient(hunting): dump: 2024-03-25T17:56:47.587 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:47.587 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:47.587 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:47.587 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:47.587 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:47.587 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:47.587 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:47.587 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:47.587 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:47.587 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:47.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.588+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3642472421 0 0) 0x7f2100136e20 con 0x7f210013f710 2024-03-25T17:56:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.588+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 >> v1:172.21.15.18:6790/0 conn(0x7f21000a98f0 legacy=0x7f21001daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.588+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 >> v1:172.21.15.169:6789/0 conn(0x7f21001430c0 legacy=0x7f21001e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.588+0000 7f20e7fff640 10 monclient: _finish_hunting 0 2024-03-25T17:56:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.588+0000 7f20e7fff640 1 monclient: found mon.a 2024-03-25T17:56:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.588+0000 7f20e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.588+0000 7f20e7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.588+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2100135330 con 0x7f210013f710 2024-03-25T17:56:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.588+0000 7f20e7fff640 10 monclient: _finish_auth 0 2024-03-25T17:56:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.588+0000 7f20e7fff640 10 monclient: _check_auth_tickets 2024-03-25T17:56:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.588+0000 7f20e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:47.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.590+0000 7f2105c8d640 5 monclient: authenticate success, global_id 4215 2024-03-25T17:56:47.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.590+0000 7f2105c8d640 10 monclient: _renew_subs 2024-03-25T17:56:47.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.590+0000 7f2105c8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:47.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.590+0000 7f2105c8d640 1 -- 172.21.15.18:0/1133448644 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2100135560 con 0x7f210013f710 2024-03-25T17:56:47.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.590+0000 7f2105c8d640 10 client.4215.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:56:47.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.590+0000 7f2105c8d640 10 monclient: _renew_subs 2024-03-25T17:56:47.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.590+0000 7f2105c8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:47.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.590+0000 7f2105c8d640 1 -- 172.21.15.18:0/1133448644 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f210013b490 con 0x7f210013f710 2024-03-25T17:56:47.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.590+0000 7f2105c8d640 1 librados: init done 2024-03-25T17:56:47.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.591+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20f4000f90 con 0x7f210013f710 2024-03-25T17:56:47.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.591+0000 7f20e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:47.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.591+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f20f4003790 con 0x7f210013f710 2024-03-25T17:56:47.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.591+0000 7f20e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:47.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.591+0000 7f20e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:47.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.591+0000 7f20e7fff640 10 monclient: dump: 2024-03-25T17:56:47.592 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:47.592 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:47.592 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:47.592 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:47.593 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:47.593 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:47.593 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:47.593 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:47.593 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:47.593 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:47.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.591+0000 7f20fdffb640 4 set_mon_vals no callback set 2024-03-25T17:56:47.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.591+0000 7f2105c8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:47.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.592+0000 7f2105c8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:47.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.592+0000 7f2105c8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:47.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.592+0000 7f2105c8d640 1 -- 172.21.15.18:0/1133448644 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e0005080 con 0x7f210013f710 2024-03-25T17:56:47.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.606+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f20f4019760 con 0x7f210013f710 2024-03-25T17:56:47.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.607+0000 7f20ff7fe640 10 monclient: get_auth_request con 0x7f20ec06e040 auth_method 0 2024-03-25T17:56:47.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.607+0000 7f20e7fff640 10 client.4215.objecter ms_handle_connect 0x7f20ec06e040 2024-03-25T17:56:47.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.609+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f20f4053470 con 0x7f210013f710 2024-03-25T17:56:47.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.609+0000 7f20e7fff640 10 client.4215.objecter ms_dispatch 0x7f2100000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:56:47.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.609+0000 7f20e7fff640 3 client.4215.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:56:47.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.609+0000 7f20e7fff640 3 client.4215.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:56:47.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.610+0000 7f20e7fff640 20 client.4215.objecter dump_active .. 0 homeless 2024-03-25T17:56:47.686 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:16.435 35734== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:47.705 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:16.464 35733== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:47.765 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:16.505 35736== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:47.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.950+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20f404edb0 con 0x7f210013f710 2024-03-25T17:56:47.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.950+0000 7f20e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:47.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:47.950+0000 7f20e7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:56:48.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.238+0000 7f2105c8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:56:48.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.238+0000 7f2105c8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:56:48.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.238+0000 7f2105c8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:56:48.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.238+0000 7f2105c8d640 1 -- 172.21.15.18:0/1133448644 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20e0005330 con 0x7f210013f710 2024-03-25T17:56:48.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.307+0000 7f20e7fff640 1 -- 172.21.15.18:0/1133448644 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f20f4050eb0 con 0x7f210013f710 2024-03-25T17:56:48.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.307+0000 7f20e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:56:48.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.307+0000 7f20e7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:56:48.307 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:56:48.307 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:56:48.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.310+0000 7f2105c8d640 10 librados: watch_flush enter 2024-03-25T17:56:48.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.310+0000 7f2105c8d640 10 librados: watch_flush exit 2024-03-25T17:56:48.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.310+0000 7f2105c8d640 20 client.4215.objecter shutdown clearing up homeless session... 2024-03-25T17:56:48.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.310+0000 7f2105c8d640 10 client.4215.objecter successfully canceled tick 2024-03-25T17:56:48.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.310+0000 7f2105c8d640 1 -- 172.21.15.18:0/1133448644 >> v1:172.21.15.18:6800/3721244451 conn(0x7f20ec06e040 legacy=0x7f20ec070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:48.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.310+0000 7f2105c8d640 10 monclient: shutdown 2024-03-25T17:56:48.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.310+0000 7f2105c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:48.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.310+0000 7f2105c8d640 1 -- 172.21.15.18:0/1133448644 >> v1:172.21.15.18:6789/0 conn(0x7f210013f710 legacy=0x7f21001db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:48.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.311+0000 7f2105c8d640 1 -- 172.21.15.18:0/1133448644 shutdown_connections 2024-03-25T17:56:48.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.311+0000 7f2105c8d640 1 -- 172.21.15.18:0/1133448644 >> 172.21.15.18:0/1133448644 conn(0x7f21000b7ea0 msgr2=0x7f2100074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:48.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.311+0000 7f2105c8d640 1 -- 172.21.15.18:0/1133448644 shutdown_connections 2024-03-25T17:56:48.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.311+0000 7f2105c8d640 1 -- 172.21.15.18:0/1133448644 wait complete. 2024-03-25T17:56:48.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:48.311+0000 7f2105c8d640 1 librados: shutdown 2024-03-25T17:56:48.324 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:56:48.366 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:17.111 35735== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:48.434 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:17.186 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:48.490 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:17.232 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:48.519 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:17.255 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:49.629 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:18.378 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:52.958 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:21.716 35733== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:52.961 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:21.721 35733== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:53.559 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:22.299 35736== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:53.563 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:22.303 35736== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:53.697 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:22.447 35734== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:53.702 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:22.451 35734== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:54.136 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:22.878 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:54.140 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:22.882 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:54.338 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:23.083 35735== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:54.342 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:23.087 35735== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:54.553 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:23.289 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:54.558 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:23.295 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:54.664 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:23.416 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:54.668 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:23.420 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:55.075 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:23.824 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:55.081 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:23.830 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:55.129 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:56:55.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.231+0000 7ff74d50e640 10 monclient: get_monmap_and_config 2024-03-25T17:56:55.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.231+0000 7ff74d50e640 10 monclient: build_initial_monmap 2024-03-25T17:56:55.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.231+0000 7ff74d50e640 10 monclient: monmap: 2024-03-25T17:56:55.232 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:56:55.232 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:55.232 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:56:55.232172+0000 2024-03-25T17:56:55.232 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:56:55.232172+0000 2024-03-25T17:56:55.232 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:55.232 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:55.232 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:55.232 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:56:55.232 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:56:55.232 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:55.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.232+0000 7ff74d50e640 1 Processor -- start 2024-03-25T17:56:55.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.232+0000 7ff74d50e640 1 -- start start 2024-03-25T17:56:55.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.232+0000 7ff74d50e640 10 monclient: init 2024-03-25T17:56:55.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.233+0000 7ff74d50e640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:55.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.233+0000 7ff74d50e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:56:55.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.233+0000 7ff74d50e640 10 monclient(hunting): picked mon.noname-c con 0x7ff7480a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:55.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.233+0000 7ff74d50e640 10 monclient(hunting): picked mon.noname-b con 0x7ff74813f710 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:55.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.233+0000 7ff74d50e640 10 monclient(hunting): picked mon.noname-a con 0x7ff7481430c0 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:55.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.233+0000 7ff74d50e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff748142520 con 0x7ff7481430c0 2024-03-25T17:56:55.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.233+0000 7ff74d50e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7480b28a0 con 0x7ff7480a98f0 2024-03-25T17:56:55.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.234+0000 7ff74d50e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7480b2a30 con 0x7ff74813f710 2024-03-25T17:56:55.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.234+0000 7ff74d50e640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:55.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.234+0000 7ff74d50e640 10 monclient(hunting): authenticate will time out at 6943.811820s 2024-03-25T17:56:55.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.234+0000 7ff7467fc640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7ff74813f710 0x7ff7481429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:40838/0 (socket says 172.21.15.18:40838) 2024-03-25T17:56:55.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.234+0000 7ff7467fc640 1 -- 172.21.15.18:0/3752180125 learned_addr learned my addr 172.21.15.18:0/3752180125 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:56:55.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 1 -- 172.21.15.18:0/3752180125 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687697362 0 0) 0x7ff7480b2a30 con 0x7ff74813f710 2024-03-25T17:56:55.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:56:55.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:55.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:55.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 10 monclient(hunting): my global_id is 4226 2024-03-25T17:56:55.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 1 -- 172.21.15.18:0/3752180125 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff730003520 con 0x7ff74813f710 2024-03-25T17:56:55.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 1 -- 172.21.15.18:0/3752180125 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223648233 0 0) 0x7ff7480b28a0 con 0x7ff7480a98f0 2024-03-25T17:56:55.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:56:55.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:55.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:55.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 10 monclient(hunting): my global_id is 4213 2024-03-25T17:56:55.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 1 -- 172.21.15.18:0/3752180125 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7480b2a30 con 0x7ff7480a98f0 2024-03-25T17:56:55.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 1 -- 172.21.15.18:0/3752180125 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222574066 0 0) 0x7ff748142520 con 0x7ff7481430c0 2024-03-25T17:56:55.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:56:55.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:55.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:55.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 10 monclient(hunting): my global_id is 4218 2024-03-25T17:56:55.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.238+0000 7ff745ffb640 1 -- 172.21.15.18:0/3752180125 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7480b28a0 con 0x7ff7481430c0 2024-03-25T17:56:55.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.242+0000 7ff745ffb640 1 -- 172.21.15.18:0/3752180125 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3819305747 0 0) 0x7ff7480b2a30 con 0x7ff7480a98f0 2024-03-25T17:56:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.242+0000 7ff745ffb640 1 -- 172.21.15.18:0/3752180125 >> v1:172.21.15.18:6790/0 conn(0x7ff74813f710 legacy=0x7ff7481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.242+0000 7ff745ffb640 1 -- 172.21.15.18:0/3752180125 >> v1:172.21.15.18:6789/0 conn(0x7ff7481430c0 legacy=0x7ff74814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.242+0000 7ff745ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:56:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.242+0000 7ff745ffb640 1 monclient: found mon.noname-c 2024-03-25T17:56:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.242+0000 7ff745ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.242+0000 7ff745ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T17:56:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.242+0000 7ff745ffb640 1 -- 172.21.15.18:0/3752180125 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7480b44c0 con 0x7ff7480a98f0 2024-03-25T17:56:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.242+0000 7ff745ffb640 10 monclient: _finish_auth 0 2024-03-25T17:56:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.242+0000 7ff745ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:56:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.242+0000 7ff745ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.243+0000 7ff74d50e640 5 monclient: authenticate success, global_id 4213 2024-03-25T17:56:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.243+0000 7ff74d50e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.243+0000 7ff745ffb640 1 -- 172.21.15.18:0/3752180125 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ff7380015b0 con 0x7ff7480a98f0 2024-03-25T17:56:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.243+0000 7ff745ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:55.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.243+0000 7ff745ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:56:55.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.243+0000 7ff745ffb640 10 monclient: dump: 2024-03-25T17:56:55.245 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:55.245 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:55.245 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:55.245 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:55.245 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:55.245 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:55.245 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:55.245 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:55.245 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:55.245 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:55.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.243+0000 7ff74d50e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:56:55.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.245+0000 7ff745ffb640 1 -- 172.21.15.18:0/3752180125 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff738001800 con 0x7ff7480a98f0 2024-03-25T17:56:55.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.245+0000 7ff745ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:55.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.245+0000 7ff745ffb640 1 -- 172.21.15.18:0/3752180125 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ff738001b90 con 0x7ff7480a98f0 2024-03-25T17:56:55.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.245+0000 7ff745ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:55.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.245+0000 7ff745ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:56:55.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.245+0000 7ff745ffb640 10 monclient: dump: 2024-03-25T17:56:55.247 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:55.247 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:55.247 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:55.248 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:55.248 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:55.248 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:55.248 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:55.248 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:55.248 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:55.248 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:55.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.245+0000 7ff74d50e640 10 monclient: get_monmap_and_config success 2024-03-25T17:56:55.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.245+0000 7ff74d50e640 4 set_mon_vals no callback set 2024-03-25T17:56:55.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.246+0000 7ff74d50e640 10 monclient: shutdown 2024-03-25T17:56:55.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.246+0000 7ff74d50e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:55.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.246+0000 7ff74d50e640 1 -- 172.21.15.18:0/3752180125 >> v1:172.21.15.169:6789/0 conn(0x7ff7480a98f0 legacy=0x7ff7480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:55.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.246+0000 7ff74d50e640 1 -- 172.21.15.18:0/3752180125 shutdown_connections 2024-03-25T17:56:55.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.246+0000 7ff74d50e640 1 -- 172.21.15.18:0/3752180125 >> 172.21.15.18:0/3752180125 conn(0x7ff7480b7ea0 msgr2=0x7ff7480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:55.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.246+0000 7ff74d50e640 1 -- 172.21.15.18:0/3752180125 shutdown_connections 2024-03-25T17:56:55.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.246+0000 7ff74d50e640 1 -- 172.21.15.18:0/3752180125 wait complete. 2024-03-25T17:56:55.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.247+0000 7ff74d50e640 10 monclient: build_initial_monmap 2024-03-25T17:56:55.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.247+0000 7ff74d50e640 10 monclient: monmap: 2024-03-25T17:56:55.249 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:56:55.249 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:56:55.249 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:56:55.249 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:56:55.249 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:56:55.249 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:56:55.249 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:56:55.249 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:56:55.250 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:56:55.250 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:55.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.248+0000 7ff74d50e640 1 librados: starting msgr at 2024-03-25T17:56:55.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.248+0000 7ff74d50e640 1 librados: starting objecter 2024-03-25T17:56:55.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.248+0000 7ff74d50e640 1 Processor -- start 2024-03-25T17:56:55.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.248+0000 7ff74d50e640 1 -- start start 2024-03-25T17:56:55.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.248+0000 7ff74d50e640 1 librados: setting wanted keys 2024-03-25T17:56:55.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.248+0000 7ff74d50e640 1 librados: calling monclient init 2024-03-25T17:56:55.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.248+0000 7ff74d50e640 10 monclient: init 2024-03-25T17:56:55.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.249+0000 7ff74d50e640 10 monclient: _reopen_session rank -1 2024-03-25T17:56:55.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.249+0000 7ff74d50e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:56:55.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.249+0000 7ff74d50e640 10 monclient(hunting): picked mon.noname-c con 0x7ff7480a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T17:56:55.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.249+0000 7ff74d50e640 10 monclient(hunting): picked mon.noname-a con 0x7ff74813f710 addr v1:172.21.15.18:6789/0 2024-03-25T17:56:55.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.250+0000 7ff74d50e640 10 monclient(hunting): picked mon.noname-b con 0x7ff7481430c0 addr v1:172.21.15.169:6789/0 2024-03-25T17:56:55.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.250+0000 7ff746ffd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7ff7480a98f0 0x7ff74813bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:40854/0 (socket says 172.21.15.18:40854) 2024-03-25T17:56:55.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.250+0000 7ff746ffd640 1 -- 172.21.15.18:0/37815226 learned_addr learned my addr 172.21.15.18:0/37815226 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:56:55.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.250+0000 7ff74d50e640 1 -- 172.21.15.18:0/37815226 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff74813a6a0 con 0x7ff74813f710 2024-03-25T17:56:55.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.250+0000 7ff74d50e640 1 -- 172.21.15.18:0/37815226 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff74813b660 con 0x7ff7481430c0 2024-03-25T17:56:55.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.250+0000 7ff74d50e640 1 -- 172.21.15.18:0/37815226 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff74813b940 con 0x7ff7480a98f0 2024-03-25T17:56:55.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.250+0000 7ff74d50e640 10 monclient(hunting): _renew_subs 2024-03-25T17:56:55.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.250+0000 7ff72f7fe640 10 client.?.objecter ms_handle_connect 0x7ff7480a98f0 2024-03-25T17:56:55.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.250+0000 7ff72f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:55.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.250+0000 7ff74d50e640 10 monclient(hunting): authenticate will time out at 6943.828593s 2024-03-25T17:56:55.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.250+0000 7ff72f7fe640 10 client.?.objecter ms_handle_connect 0x7ff74813f710 2024-03-25T17:56:55.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.251+0000 7ff72f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:55.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.251+0000 7ff72f7fe640 10 client.?.objecter ms_handle_connect 0x7ff7481430c0 2024-03-25T17:56:55.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.251+0000 7ff72f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:56:55.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977683388 0 0) 0x7ff74813b940 con 0x7ff7480a98f0 2024-03-25T17:56:55.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:56:55.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:55.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:55.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 10 monclient(hunting): my global_id is 4229 2024-03-25T17:56:55.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff734004eb0 con 0x7ff7480a98f0 2024-03-25T17:56:55.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515244435 0 0) 0x7ff74813a6a0 con 0x7ff74813f710 2024-03-25T17:56:55.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:56:55.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:55.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:55.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 10 monclient(hunting): my global_id is 4221 2024-03-25T17:56:55.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff74813b940 con 0x7ff74813f710 2024-03-25T17:56:55.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599647850 0 0) 0x7ff74813b660 con 0x7ff7481430c0 2024-03-25T17:56:55.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:56:55.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:56:55.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:56:55.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 10 monclient(hunting): my global_id is 4216 2024-03-25T17:56:55.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.253+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff734005360 con 0x7ff7481430c0 2024-03-25T17:56:55.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.259+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3812601100 0 0) 0x7ff734004eb0 con 0x7ff7480a98f0 2024-03-25T17:56:55.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.259+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff74813b660 con 0x7ff7480a98f0 2024-03-25T17:56:55.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.260+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1225328258 0 0) 0x7ff734005360 con 0x7ff7481430c0 2024-03-25T17:56:55.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.260+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff734004eb0 con 0x7ff7481430c0 2024-03-25T17:56:55.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.260+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ff7380035f0 con 0x7ff7480a98f0 2024-03-25T17:56:55.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.260+0000 7ff72f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:55.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.260+0000 7ff72f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:56:55.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.260+0000 7ff72f7fe640 10 monclient(hunting): dump: 2024-03-25T17:56:55.261 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:55.261 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:55.261 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:55.261 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:55.262 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:55.262 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:55.262 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:55.262 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:55.262 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:55.262 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:55.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.260+0000 7ff72f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:56:55.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.260+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ff740001bd0 con 0x7ff7481430c0 2024-03-25T17:56:55.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.260+0000 7ff72f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:55.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.260+0000 7ff72f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:56:55.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.260+0000 7ff72f7fe640 10 monclient(hunting): dump: 2024-03-25T17:56:55.262 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:55.262 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:55.263 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:55.263 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:55.263 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:55.263 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:55.263 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:55.263 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:55.263 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:55.263 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:55.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.260+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3779737675 0 0) 0x7ff74813b940 con 0x7ff74813f710 2024-03-25T17:56:55.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.260+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff734006150 con 0x7ff74813f710 2024-03-25T17:56:55.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.260+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ff73c001950 con 0x7ff74813f710 2024-03-25T17:56:55.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.260+0000 7ff72f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:55.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.260+0000 7ff72f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:56:55.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.260+0000 7ff72f7fe640 10 monclient(hunting): dump: 2024-03-25T17:56:55.264 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:55.264 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:55.264 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:55.264 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:55.264 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:55.264 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:55.264 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:55.264 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:55.264 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:55.264 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:55.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.264+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3077323777 0 0) 0x7ff734004eb0 con 0x7ff7481430c0 2024-03-25T17:56:55.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.264+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 >> v1:172.21.15.18:6790/0 conn(0x7ff7480a98f0 legacy=0x7ff74813bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:55.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.264+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 >> v1:172.21.15.18:6789/0 conn(0x7ff74813f710 legacy=0x7ff7480b7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:55.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.264+0000 7ff72f7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:56:55.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.264+0000 7ff72f7fe640 1 monclient: found mon.b 2024-03-25T17:56:55.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.264+0000 7ff72f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:56:55.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.265+0000 7ff72f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:55.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.265+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74813a1e0 con 0x7ff7481430c0 2024-03-25T17:56:55.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.265+0000 7ff72f7fe640 10 monclient: _finish_auth 0 2024-03-25T17:56:55.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.265+0000 7ff72f7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:56:55.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.265+0000 7ff72f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:56:55.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.265+0000 7ff74d50e640 5 monclient: authenticate success, global_id 4216 2024-03-25T17:56:55.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.265+0000 7ff74d50e640 10 monclient: _renew_subs 2024-03-25T17:56:55.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.265+0000 7ff74d50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:55.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.265+0000 7ff74d50e640 1 -- 172.21.15.18:0/37815226 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff74812dcf0 con 0x7ff7481430c0 2024-03-25T17:56:55.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.265+0000 7ff74d50e640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:56:55.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.265+0000 7ff74d50e640 10 monclient: _renew_subs 2024-03-25T17:56:55.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.265+0000 7ff74d50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:55.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.265+0000 7ff74d50e640 1 -- 172.21.15.18:0/37815226 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff74812f880 con 0x7ff7481430c0 2024-03-25T17:56:55.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.265+0000 7ff74d50e640 1 librados: init done 2024-03-25T17:56:55.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.266+0000 7ff74d50e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:55.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.266+0000 7ff74d50e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:55.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.266+0000 7ff74d50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:55.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.266+0000 7ff74d50e640 1 -- 172.21.15.18:0/37815226 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff74814f2d0 con 0x7ff7481430c0 2024-03-25T17:56:55.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.266+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff740001fe0 con 0x7ff7481430c0 2024-03-25T17:56:55.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.266+0000 7ff72f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:56:55.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.266+0000 7ff7457fa640 4 set_mon_vals no callback set 2024-03-25T17:56:55.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.564+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ff740001bd0 con 0x7ff7481430c0 2024-03-25T17:56:55.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.564+0000 7ff72f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:56:55.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.564+0000 7ff72f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:56:55.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.564+0000 7ff72f7fe640 10 monclient: dump: 2024-03-25T17:56:55.564 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:56:55.564 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:56:55.564 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:55.564 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:56:55.564 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:56:55.564 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:56:55.564 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:56:55.564 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:56:55.565 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:56:55.565 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:56:55.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.565+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7ff740003990 con 0x7ff7481430c0 2024-03-25T17:56:55.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.565+0000 7ff746ffd640 10 monclient: get_auth_request con 0x7ff73406e750 auth_method 0 2024-03-25T17:56:55.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.566+0000 7ff72f7fe640 10 client.4216.objecter ms_handle_connect 0x7ff73406e750 2024-03-25T17:56:55.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.567+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7ff740053450 con 0x7ff7481430c0 2024-03-25T17:56:55.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.567+0000 7ff72f7fe640 10 client.4216.objecter ms_dispatch 0x7ff748000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:56:55.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.567+0000 7ff72f7fe640 3 client.4216.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:56:55.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.567+0000 7ff72f7fe640 3 client.4216.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:56:55.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.567+0000 7ff72f7fe640 20 client.4216.objecter dump_active .. 0 homeless 2024-03-25T17:56:55.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.570+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff74004ef60 con 0x7ff7481430c0 2024-03-25T17:56:55.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.570+0000 7ff72f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:56:55.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.570+0000 7ff72f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:56:55.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.809+0000 7ff74d50e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:56:55.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.809+0000 7ff74d50e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:56:55.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.809+0000 7ff74d50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:56:55.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.809+0000 7ff74d50e640 1 -- 172.21.15.18:0/37815226 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7480627b0 con 0x7ff7481430c0 2024-03-25T17:56:55.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.834+0000 7ff72f7fe640 1 -- 172.21.15.18:0/37815226 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7ff740050e90 con 0x7ff7481430c0 2024-03-25T17:56:55.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.834+0000 7ff72f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:56:55.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.834+0000 7ff72f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:56:55.835 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:56:55.835 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:56:55.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.837+0000 7ff74d50e640 10 librados: watch_flush enter 2024-03-25T17:56:55.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.837+0000 7ff74d50e640 10 librados: watch_flush exit 2024-03-25T17:56:55.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.837+0000 7ff74d50e640 20 client.4216.objecter shutdown clearing up homeless session... 2024-03-25T17:56:55.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.837+0000 7ff74d50e640 10 client.4216.objecter successfully canceled tick 2024-03-25T17:56:55.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.837+0000 7ff74d50e640 1 -- 172.21.15.18:0/37815226 >> v1:172.21.15.18:6800/3721244451 conn(0x7ff73406e750 legacy=0x7ff734070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:55.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.837+0000 7ff74d50e640 10 monclient: shutdown 2024-03-25T17:56:55.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.837+0000 7ff74d50e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:56:55.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.837+0000 7ff74d50e640 1 -- 172.21.15.18:0/37815226 >> v1:172.21.15.169:6789/0 conn(0x7ff7481430c0 legacy=0x7ff7481e2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:56:55.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.837+0000 7ff74d50e640 1 -- 172.21.15.18:0/37815226 shutdown_connections 2024-03-25T17:56:55.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.837+0000 7ff74d50e640 1 -- 172.21.15.18:0/37815226 >> 172.21.15.18:0/37815226 conn(0x7ff7480b7ea0 msgr2=0x7ff748074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:56:55.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.837+0000 7ff74d50e640 1 -- 172.21.15.18:0/37815226 shutdown_connections 2024-03-25T17:56:55.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.837+0000 7ff74d50e640 1 -- 172.21.15.18:0/37815226 wait complete. 2024-03-25T17:56:55.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:56:55.837+0000 7ff74d50e640 1 librados: shutdown 2024-03-25T17:56:55.847 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:56:57.163 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:00:25.922 35733== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:57.753 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:00:26.503 35734== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:57.770 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:00:26.510 35736== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:58.495 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:00:27.237 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:58.852 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:00:27.597 35735== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:59.007 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:00:27.759 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:59.308 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:00:28.044 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T17:56:59.576 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:00:28.325 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T17:57:02.655 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:57:02.745 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.746+0000 7f5ee7765640 10 monclient: get_monmap_and_config 2024-03-25T17:57:02.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.746+0000 7f5ee7765640 10 monclient: build_initial_monmap 2024-03-25T17:57:02.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.746+0000 7f5ee7765640 10 monclient: monmap: 2024-03-25T17:57:02.747 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:57:02.747 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:02.747 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:57:02.746644+0000 2024-03-25T17:57:02.747 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:57:02.746644+0000 2024-03-25T17:57:02.747 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:57:02.748 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:02.748 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:57:02.748 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:57:02.748 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:57:02.748 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:02.752 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.746+0000 7f5ee7765640 1 Processor -- start 2024-03-25T17:57:02.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.747+0000 7f5ee7765640 1 -- start start 2024-03-25T17:57:02.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.747+0000 7f5ee7765640 10 monclient: init 2024-03-25T17:57:02.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.747+0000 7f5ee7765640 10 monclient: _reopen_session rank -1 2024-03-25T17:57:02.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.747+0000 7f5ee7765640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:57:02.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.747+0000 7f5ee7765640 10 monclient(hunting): picked mon.noname-c con 0x7f5ee00b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T17:57:02.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.747+0000 7f5ee7765640 10 monclient(hunting): picked mon.noname-b con 0x7f5ee00b1730 addr v1:172.21.15.18:6790/0 2024-03-25T17:57:02.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.747+0000 7f5ee7765640 10 monclient(hunting): picked mon.noname-a con 0x7f5ee014be90 addr v1:172.21.15.18:6789/0 2024-03-25T17:57:02.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.747+0000 7f5ee7765640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee00af940 con 0x7f5ee014be90 2024-03-25T17:57:02.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.747+0000 7f5ee7765640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee00b0a70 con 0x7f5ee00b4d60 2024-03-25T17:57:02.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.747+0000 7f5ee7765640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee00b0d50 con 0x7f5ee00b1730 2024-03-25T17:57:02.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.747+0000 7f5ee7765640 10 monclient(hunting): _renew_subs 2024-03-25T17:57:02.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.747+0000 7f5ee7765640 10 monclient(hunting): authenticate will time out at 6951.325480s 2024-03-25T17:57:02.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.748+0000 7f5ee4d00640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f5ee00b1730 0x7f5ee00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:40864/0 (socket says 172.21.15.18:40864) 2024-03-25T17:57:02.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.748+0000 7f5ee4d00640 1 -- 172.21.15.18:0/718921377 learned_addr learned my addr 172.21.15.18:0/718921377 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:57:02.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 1 -- 172.21.15.18:0/718921377 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035523864 0 0) 0x7f5ee00b0a70 con 0x7f5ee00b4d60 2024-03-25T17:57:02.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:57:02.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:02.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:02.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 10 monclient(hunting): my global_id is 4219 2024-03-25T17:57:02.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 1 -- 172.21.15.18:0/718921377 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ecc003520 con 0x7f5ee00b4d60 2024-03-25T17:57:02.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 1 -- 172.21.15.18:0/718921377 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544059239 0 0) 0x7f5ee00b0d50 con 0x7f5ee00b1730 2024-03-25T17:57:02.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:57:02.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:02.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:02.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 10 monclient(hunting): my global_id is 4232 2024-03-25T17:57:02.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 1 -- 172.21.15.18:0/718921377 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee00b0a70 con 0x7f5ee00b1730 2024-03-25T17:57:02.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 1 -- 172.21.15.18:0/718921377 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013817160 0 0) 0x7f5ee00af940 con 0x7f5ee014be90 2024-03-25T17:57:02.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:57:02.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:02.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:02.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 10 monclient(hunting): my global_id is 4224 2024-03-25T17:57:02.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.751+0000 7f5ed7fff640 1 -- 172.21.15.18:0/718921377 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee00b0d50 con 0x7f5ee014be90 2024-03-25T17:57:02.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.754+0000 7f5ed7fff640 1 -- 172.21.15.18:0/718921377 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 378950407 0 0) 0x7f5ee00b0a70 con 0x7f5ee00b1730 2024-03-25T17:57:02.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.754+0000 7f5ed7fff640 1 -- 172.21.15.18:0/718921377 >> v1:172.21.15.169:6789/0 conn(0x7f5ee00b4d60 legacy=0x7f5ee013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:02.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.755+0000 7f5ed7fff640 1 -- 172.21.15.18:0/718921377 >> v1:172.21.15.18:6789/0 conn(0x7f5ee014be90 legacy=0x7f5ee014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:02.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.755+0000 7f5ed7fff640 10 monclient: _finish_hunting 0 2024-03-25T17:57:02.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.755+0000 7f5ed7fff640 1 monclient: found mon.noname-b 2024-03-25T17:57:02.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.755+0000 7f5ed7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:57:02.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.755+0000 7f5ed7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T17:57:02.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.755+0000 7f5ed7fff640 1 -- 172.21.15.18:0/718921377 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee00b1030 con 0x7f5ee00b1730 2024-03-25T17:57:02.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.755+0000 7f5ed7fff640 10 monclient: _finish_auth 0 2024-03-25T17:57:02.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.755+0000 7f5ed7fff640 10 monclient: _check_auth_tickets 2024-03-25T17:57:02.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.755+0000 7f5ed7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:57:02.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.755+0000 7f5ed7fff640 1 -- 172.21.15.18:0/718921377 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5ed0001670 con 0x7f5ee00b1730 2024-03-25T17:57:02.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.755+0000 7f5ed7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:02.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.755+0000 7f5ed7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:57:02.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.755+0000 7f5ed7fff640 10 monclient: dump: 2024-03-25T17:57:02.764 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:02.764 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:02.764 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:02.764 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:02.764 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:02.764 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:02.764 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:02.764 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:02.764 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:02.764 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:02.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.755+0000 7f5ee7765640 5 monclient: authenticate success, global_id 4232 2024-03-25T17:57:02.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.755+0000 7f5ee7765640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:57:02.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.756+0000 7f5ed7fff640 1 -- 172.21.15.18:0/718921377 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ed00017d0 con 0x7f5ee00b1730 2024-03-25T17:57:02.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.756+0000 7f5ed7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:57:02.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.756+0000 7f5ed7fff640 1 -- 172.21.15.18:0/718921377 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5ed0001be0 con 0x7f5ee00b1730 2024-03-25T17:57:02.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.756+0000 7f5ed7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:02.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.756+0000 7f5ed7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:57:02.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.756+0000 7f5ed7fff640 10 monclient: dump: 2024-03-25T17:57:02.765 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:02.765 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:02.765 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:02.765 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:02.765 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:02.766 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:02.766 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:02.766 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:02.766 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:02.766 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:02.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.756+0000 7f5ee7765640 10 monclient: get_monmap_and_config success 2024-03-25T17:57:02.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.756+0000 7f5ee7765640 4 set_mon_vals no callback set 2024-03-25T17:57:02.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.757+0000 7f5ee7765640 10 monclient: shutdown 2024-03-25T17:57:02.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.757+0000 7f5ee7765640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:57:02.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.757+0000 7f5ee7765640 1 -- 172.21.15.18:0/718921377 >> v1:172.21.15.18:6790/0 conn(0x7f5ee00b1730 legacy=0x7f5ee00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:02.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.757+0000 7f5ee7765640 1 -- 172.21.15.18:0/718921377 shutdown_connections 2024-03-25T17:57:02.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.757+0000 7f5ee7765640 1 -- 172.21.15.18:0/718921377 >> 172.21.15.18:0/718921377 conn(0x7f5ee00617a0 msgr2=0x7f5ee0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:57:02.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.757+0000 7f5ee7765640 1 -- 172.21.15.18:0/718921377 shutdown_connections 2024-03-25T17:57:02.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.758+0000 7f5ee7765640 1 -- 172.21.15.18:0/718921377 wait complete. 2024-03-25T17:57:02.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.758+0000 7f5ee7765640 10 monclient: build_initial_monmap 2024-03-25T17:57:02.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.758+0000 7f5ee7765640 10 monclient: monmap: 2024-03-25T17:57:02.767 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:57:02.767 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:57:02.767 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:57:02.767 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:57:02.767 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:57:02.767 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:57:02.769 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:57:02.770 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:57:02.770 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:57:02.770 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:02.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.759+0000 7f5ee7765640 1 librados: starting msgr at 2024-03-25T17:57:02.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.759+0000 7f5ee7765640 1 librados: starting objecter 2024-03-25T17:57:02.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.759+0000 7f5ee7765640 1 Processor -- start 2024-03-25T17:57:02.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.759+0000 7f5ee7765640 1 -- start start 2024-03-25T17:57:02.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.759+0000 7f5ee7765640 1 librados: setting wanted keys 2024-03-25T17:57:02.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.759+0000 7f5ee7765640 1 librados: calling monclient init 2024-03-25T17:57:02.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.759+0000 7f5ee7765640 10 monclient: init 2024-03-25T17:57:02.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.759+0000 7f5ee7765640 10 monclient: _reopen_session rank -1 2024-03-25T17:57:02.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.759+0000 7f5ee7765640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:57:02.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.759+0000 7f5ee7765640 10 monclient(hunting): picked mon.noname-a con 0x7f5ee00b1730 addr v1:172.21.15.18:6789/0 2024-03-25T17:57:02.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.760+0000 7f5ee7765640 10 monclient(hunting): picked mon.noname-b con 0x7f5ee00b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T17:57:02.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.760+0000 7f5ee7765640 10 monclient(hunting): picked mon.noname-c con 0x7f5ee014be90 addr v1:172.21.15.18:6790/0 2024-03-25T17:57:02.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.760+0000 7f5ee7765640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee00a9610 con 0x7f5ee00b1730 2024-03-25T17:57:02.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.760+0000 7f5ee7765640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee01433d0 con 0x7f5ee00b4d60 2024-03-25T17:57:02.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.760+0000 7f5ee5501640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f5ee00b1730 0x7f5ee01d79a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:55556/0 (socket says 172.21.15.18:55556) 2024-03-25T17:57:02.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.760+0000 7f5ee5501640 1 -- 172.21.15.18:0/2410566384 learned_addr learned my addr 172.21.15.18:0/2410566384 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:57:02.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.760+0000 7f5ee7765640 1 -- 172.21.15.18:0/2410566384 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ee0143560 con 0x7f5ee014be90 2024-03-25T17:57:02.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.760+0000 7f5ee7765640 10 monclient(hunting): _renew_subs 2024-03-25T17:57:02.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.760+0000 7f5ee7765640 10 monclient(hunting): authenticate will time out at 6951.338062s 2024-03-25T17:57:02.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.760+0000 7f5ed5ffb640 10 client.?.objecter ms_handle_connect 0x7f5ee00b1730 2024-03-25T17:57:02.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.760+0000 7f5ed5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:02.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.761+0000 7f5ed5ffb640 10 client.?.objecter ms_handle_connect 0x7f5ee014be90 2024-03-25T17:57:02.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.761+0000 7f5ed5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:02.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.761+0000 7f5ed5ffb640 10 client.?.objecter ms_handle_connect 0x7f5ee00b4d60 2024-03-25T17:57:02.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.761+0000 7f5ed5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:02.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.763+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022990876 0 0) 0x7f5ee00a9610 con 0x7f5ee00b1730 2024-03-25T17:57:02.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.763+0000 7f5ed5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:57:02.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.763+0000 7f5ed5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:02.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.763+0000 7f5ed5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:02.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.763+0000 7f5ed5ffb640 10 monclient(hunting): my global_id is 4227 2024-03-25T17:57:02.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.763+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ec4004eb0 con 0x7f5ee00b1730 2024-03-25T17:57:02.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.763+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411220533 0 0) 0x7f5ee0143560 con 0x7f5ee014be90 2024-03-25T17:57:02.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.763+0000 7f5ed5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:57:02.773 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.763+0000 7f5ed5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:02.773 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.763+0000 7f5ed5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:02.773 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.763+0000 7f5ed5ffb640 10 monclient(hunting): my global_id is 4235 2024-03-25T17:57:02.773 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.763+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee00a9610 con 0x7f5ee014be90 2024-03-25T17:57:02.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.764+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058526844 0 0) 0x7f5ee01433d0 con 0x7f5ee00b4d60 2024-03-25T17:57:02.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.764+0000 7f5ed5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:57:02.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.764+0000 7f5ed5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:02.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.764+0000 7f5ed5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:02.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.764+0000 7f5ed5ffb640 10 monclient(hunting): my global_id is 4222 2024-03-25T17:57:02.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.764+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee0143560 con 0x7f5ee00b4d60 2024-03-25T17:57:02.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.768+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4082333927 0 0) 0x7f5ec4004eb0 con 0x7f5ee00b1730 2024-03-25T17:57:02.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.768+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ee01433d0 con 0x7f5ee00b1730 2024-03-25T17:57:02.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.768+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5edc001990 con 0x7f5ee00b1730 2024-03-25T17:57:02.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.768+0000 7f5ed5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:02.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.768+0000 7f5ed5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:57:02.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.768+0000 7f5ed5ffb640 10 monclient(hunting): dump: 2024-03-25T17:57:02.777 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:02.777 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:02.777 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:02.777 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:02.777 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:02.777 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:02.777 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:02.777 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:02.777 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:02.777 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:02.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.768+0000 7f5ed5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:57:02.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.768+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4286697325 0 0) 0x7f5ee00a9610 con 0x7f5ee014be90 2024-03-25T17:57:02.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.768+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ec4004eb0 con 0x7f5ee014be90 2024-03-25T17:57:02.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.768+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5ed8001bd0 con 0x7f5ee014be90 2024-03-25T17:57:02.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.768+0000 7f5ed5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:02.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.768+0000 7f5ed5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:57:02.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.768+0000 7f5ed5ffb640 10 monclient(hunting): dump: 2024-03-25T17:57:02.778 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:02.778 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:02.778 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:02.778 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:02.778 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:02.779 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:02.779 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:02.779 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:02.779 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:02.779 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:02.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.772+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 754595265 0 0) 0x7f5ee0143560 con 0x7f5ee00b4d60 2024-03-25T17:57:02.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.772+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ee00a9610 con 0x7f5ee00b4d60 2024-03-25T17:57:02.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.772+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5ed0001ae0 con 0x7f5ee00b4d60 2024-03-25T17:57:02.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.772+0000 7f5ed5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:02.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.772+0000 7f5ed5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:57:02.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.772+0000 7f5ed5ffb640 10 monclient(hunting): dump: 2024-03-25T17:57:02.780 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:02.780 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:02.780 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:02.780 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:02.780 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:02.780 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:02.780 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:02.780 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:02.780 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:02.780 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:02.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.773+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1927480637 0 0) 0x7f5ec4004eb0 con 0x7f5ee014be90 2024-03-25T17:57:02.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.773+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 >> v1:172.21.15.169:6789/0 conn(0x7f5ee00b4d60 legacy=0x7f5ee01d80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:02.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.773+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 >> v1:172.21.15.18:6789/0 conn(0x7f5ee00b1730 legacy=0x7f5ee01d79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:02.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.773+0000 7f5ed5ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:57:02.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.773+0000 7f5ed5ffb640 1 monclient: found mon.c 2024-03-25T17:57:02.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.773+0000 7f5ed5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:57:02.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.773+0000 7f5ed5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:02.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.773+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee01436f0 con 0x7f5ee014be90 2024-03-25T17:57:02.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.773+0000 7f5ed5ffb640 10 monclient: _finish_auth 0 2024-03-25T17:57:02.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.773+0000 7f5ed5ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:57:02.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.773+0000 7f5ed5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:57:02.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.773+0000 7f5ee7765640 5 monclient: authenticate success, global_id 4235 2024-03-25T17:57:02.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.773+0000 7f5ee7765640 10 monclient: _renew_subs 2024-03-25T17:57:02.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.773+0000 7f5ee7765640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:02.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.773+0000 7f5ee7765640 1 -- 172.21.15.18:0/2410566384 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee01429a0 con 0x7f5ee014be90 2024-03-25T17:57:02.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.774+0000 7f5ee7765640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:57:02.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.774+0000 7f5ee7765640 10 monclient: _renew_subs 2024-03-25T17:57:02.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.774+0000 7f5ee7765640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:02.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.774+0000 7f5ee7765640 1 -- 172.21.15.18:0/2410566384 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee0140380 con 0x7f5ee014be90 2024-03-25T17:57:02.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.774+0000 7f5ee7765640 1 librados: init done 2024-03-25T17:57:02.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.775+0000 7f5ee7765640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:02.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.775+0000 7f5ee7765640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:02.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.775+0000 7f5ee7765640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:02.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.775+0000 7f5ee7765640 1 -- 172.21.15.18:0/2410566384 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee0061d10 con 0x7f5ee014be90 2024-03-25T17:57:02.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.988+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ed8002020 con 0x7f5ee014be90 2024-03-25T17:57:02.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.988+0000 7f5ed5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:57:02.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:02.988+0000 7f5ed77fe640 4 set_mon_vals no callback set 2024-03-25T17:57:03.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.051+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5ed8003b30 con 0x7f5ee014be90 2024-03-25T17:57:03.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.051+0000 7f5ed5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:03.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.051+0000 7f5ed5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:57:03.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.051+0000 7f5ed5ffb640 10 monclient: dump: 2024-03-25T17:57:03.052 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:03.052 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:03.052 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:03.052 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:03.052 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:03.052 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:03.052 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:03.052 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:03.052 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:03.052 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:03.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.052+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f5ed8019b00 con 0x7f5ee014be90 2024-03-25T17:57:03.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.053+0000 7f5ee5501640 10 monclient: get_auth_request con 0x7f5ec406e110 auth_method 0 2024-03-25T17:57:03.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.053+0000 7f5ed5ffb640 10 client.4235.objecter ms_handle_connect 0x7f5ec406e110 2024-03-25T17:57:03.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.059+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f5ed80537b0 con 0x7f5ee014be90 2024-03-25T17:57:03.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.059+0000 7f5ed5ffb640 10 client.4235.objecter ms_dispatch 0x7f5ee0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:57:03.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.059+0000 7f5ed5ffb640 3 client.4235.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:57:03.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.059+0000 7f5ed5ffb640 3 client.4235.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:57:03.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.059+0000 7f5ed5ffb640 20 client.4235.objecter dump_active .. 0 homeless 2024-03-25T17:57:03.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.059+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ed804f0f0 con 0x7f5ee014be90 2024-03-25T17:57:03.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.059+0000 7f5ed5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:03.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.059+0000 7f5ed5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:57:03.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.267+0000 7f5ee7765640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:03.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.267+0000 7f5ee7765640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:03.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.267+0000 7f5ee7765640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:03.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.267+0000 7f5ee7765640 1 -- 172.21.15.18:0/2410566384 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ee00aaa20 con 0x7f5ee014be90 2024-03-25T17:57:03.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.283+0000 7f5ed5ffb640 1 -- 172.21.15.18:0/2410566384 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f5ed80511f0 con 0x7f5ee014be90 2024-03-25T17:57:03.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.283+0000 7f5ed5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:03.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.283+0000 7f5ed5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:57:03.283 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:57:03.284 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:57:03.285 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.285+0000 7f5ee7765640 10 librados: watch_flush enter 2024-03-25T17:57:03.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.286+0000 7f5ee7765640 10 librados: watch_flush exit 2024-03-25T17:57:03.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.286+0000 7f5ee7765640 20 client.4235.objecter shutdown clearing up homeless session... 2024-03-25T17:57:03.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.286+0000 7f5ee7765640 10 client.4235.objecter successfully canceled tick 2024-03-25T17:57:03.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.286+0000 7f5ee7765640 1 -- 172.21.15.18:0/2410566384 >> v1:172.21.15.18:6800/3721244451 conn(0x7f5ec406e110 legacy=0x7f5ec40705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:03.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.286+0000 7f5ee7765640 10 monclient: shutdown 2024-03-25T17:57:03.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.286+0000 7f5ee7765640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:57:03.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.286+0000 7f5ee7765640 1 -- 172.21.15.18:0/2410566384 >> v1:172.21.15.18:6790/0 conn(0x7f5ee014be90 legacy=0x7f5ee01e2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:03.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.286+0000 7f5ee7765640 1 -- 172.21.15.18:0/2410566384 shutdown_connections 2024-03-25T17:57:03.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.286+0000 7f5ee7765640 1 -- 172.21.15.18:0/2410566384 >> 172.21.15.18:0/2410566384 conn(0x7f5ee00617a0 msgr2=0x7f5ee013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:57:03.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.287+0000 7f5ee7765640 1 -- 172.21.15.18:0/2410566384 shutdown_connections 2024-03-25T17:57:03.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.287+0000 7f5ee7765640 1 -- 172.21.15.18:0/2410566384 wait complete. 2024-03-25T17:57:03.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:03.287+0000 7f5ee7765640 1 librados: shutdown 2024-03-25T17:57:03.297 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:57:05.943 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-03-25T17:57:05.940+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T17:57:06.312 INFO:tasks.ceph.osd.7.smithi169.stderr:2024-03-25T17:57:06.309+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T17:57:06.688 INFO:tasks.ceph.osd.5.smithi169.stderr:2024-03-25T17:57:06.678+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T17:57:07.127 INFO:tasks.ceph.osd.2.smithi018.stderr:2024-03-25T17:57:07.124+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T17:57:07.342 INFO:tasks.ceph.osd.4.smithi169.stderr:2024-03-25T17:57:07.338+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T17:57:07.733 INFO:tasks.ceph.osd.1.smithi018.stderr:2024-03-25T17:57:07.730+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T17:57:07.887 INFO:tasks.ceph.osd.3.smithi018.stderr:2024-03-25T17:57:07.884+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T17:57:08.509 INFO:tasks.ceph.osd.0.smithi018.stderr:2024-03-25T17:57:08.505+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T17:57:10.104 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:57:10.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.205+0000 7f02f8eff640 10 monclient: get_monmap_and_config 2024-03-25T17:57:10.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.205+0000 7f02f8eff640 10 monclient: build_initial_monmap 2024-03-25T17:57:10.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.205+0000 7f02f8eff640 10 monclient: monmap: 2024-03-25T17:57:10.206 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:57:10.206 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:10.206 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:57:10.206931+0000 2024-03-25T17:57:10.206 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:57:10.206931+0000 2024-03-25T17:57:10.206 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:57:10.206 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:10.207 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:57:10.207 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:57:10.207 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:57:10.207 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:10.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.206+0000 7f02f8eff640 1 Processor -- start 2024-03-25T17:57:10.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.206+0000 7f02f8eff640 1 -- start start 2024-03-25T17:57:10.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.206+0000 7f02f8eff640 10 monclient: init 2024-03-25T17:57:10.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.207+0000 7f02f8eff640 10 monclient: _reopen_session rank -1 2024-03-25T17:57:10.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.207+0000 7f02f8eff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:57:10.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.207+0000 7f02f8eff640 10 monclient(hunting): picked mon.noname-a con 0x7f02f40b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T17:57:10.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.207+0000 7f02f8eff640 10 monclient(hunting): picked mon.noname-c con 0x7f02f40b1730 addr v1:172.21.15.169:6789/0 2024-03-25T17:57:10.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.207+0000 7f02f8eff640 10 monclient(hunting): picked mon.noname-b con 0x7f02f414be90 addr v1:172.21.15.18:6790/0 2024-03-25T17:57:10.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.207+0000 7f02f8eff640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f40af940 con 0x7f02f40b4d60 2024-03-25T17:57:10.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.207+0000 7f02f8eff640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f40b0a70 con 0x7f02f40b1730 2024-03-25T17:57:10.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.207+0000 7f02f8eff640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f40b0d50 con 0x7f02f414be90 2024-03-25T17:57:10.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.207+0000 7f02f8eff640 10 monclient(hunting): _renew_subs 2024-03-25T17:57:10.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.207+0000 7f02f8eff640 10 monclient(hunting): authenticate will time out at 6958.785752s 2024-03-25T17:57:10.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.207+0000 7f02f2d9d640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f02f414be90 0x7f02f414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:57792/0 (socket says 172.21.15.18:57792) 2024-03-25T17:57:10.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.207+0000 7f02f2d9d640 1 -- 172.21.15.18:0/965959589 learned_addr learned my addr 172.21.15.18:0/965959589 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:57:10.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.217+0000 7f02f1d9b640 1 -- 172.21.15.18:0/965959589 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709252960 0 0) 0x7f02f40af940 con 0x7f02f40b4d60 2024-03-25T17:57:10.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.217+0000 7f02f1d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:57:10.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.217+0000 7f02f1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:10.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.217+0000 7f02f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:10.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.217+0000 7f02f1d9b640 10 monclient(hunting): my global_id is 4254 2024-03-25T17:57:10.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.217+0000 7f02f1d9b640 1 -- 172.21.15.18:0/965959589 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e0003520 con 0x7f02f40b4d60 2024-03-25T17:57:10.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.219+0000 7f02f1d9b640 1 -- 172.21.15.18:0/965959589 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724774139 0 0) 0x7f02f40b0d50 con 0x7f02f414be90 2024-03-25T17:57:10.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.219+0000 7f02f1d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:57:10.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.219+0000 7f02f1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:10.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.219+0000 7f02f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:10.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.219+0000 7f02f1d9b640 10 monclient(hunting): my global_id is 4262 2024-03-25T17:57:10.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.219+0000 7f02f1d9b640 1 -- 172.21.15.18:0/965959589 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f40af940 con 0x7f02f414be90 2024-03-25T17:57:10.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f1d9b640 1 -- 172.21.15.18:0/965959589 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1420505492 0 0) 0x7f02e0003520 con 0x7f02f40b4d60 2024-03-25T17:57:10.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f1d9b640 1 -- 172.21.15.18:0/965959589 >> v1:172.21.15.18:6790/0 conn(0x7f02f414be90 legacy=0x7f02f414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:10.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f1d9b640 1 -- 172.21.15.18:0/965959589 >> v1:172.21.15.169:6789/0 conn(0x7f02f40b1730 legacy=0x7f02f40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:10.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f1d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:57:10.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f1d9b640 1 monclient: found mon.noname-a 2024-03-25T17:57:10.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:57:10.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T17:57:10.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f1d9b640 1 -- 172.21.15.18:0/965959589 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f40b1030 con 0x7f02f40b4d60 2024-03-25T17:57:10.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f1d9b640 10 monclient: _finish_auth 0 2024-03-25T17:57:10.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f1d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:57:10.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:57:10.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f1d9b640 1 -- 172.21.15.18:0/965959589 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f02e4001690 con 0x7f02f40b4d60 2024-03-25T17:57:10.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f8eff640 5 monclient: authenticate success, global_id 4254 2024-03-25T17:57:10.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f8eff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:57:10.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:10.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:57:10.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f1d9b640 10 monclient: dump: 2024-03-25T17:57:10.241 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:10.241 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:10.241 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:10.241 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:10.241 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:10.241 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:10.241 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:10.242 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:10.242 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:10.242 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:10.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.239+0000 7f02f8eff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:57:10.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.257+0000 7f02f1d9b640 1 -- 172.21.15.18:0/965959589 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02e40017f0 con 0x7f02f40b4d60 2024-03-25T17:57:10.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.257+0000 7f02f1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:57:10.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.257+0000 7f02f1d9b640 1 -- 172.21.15.18:0/965959589 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f02e4001c00 con 0x7f02f40b4d60 2024-03-25T17:57:10.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.257+0000 7f02f8eff640 10 monclient: get_monmap_and_config success 2024-03-25T17:57:10.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.257+0000 7f02f8eff640 4 set_mon_vals no callback set 2024-03-25T17:57:10.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.257+0000 7f02f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:10.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.257+0000 7f02f1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:57:10.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.257+0000 7f02f1d9b640 10 monclient: dump: 2024-03-25T17:57:10.258 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:10.258 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:10.258 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:10.258 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:10.258 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:10.258 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:10.259 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:10.259 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:10.259 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:10.259 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:10.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.258+0000 7f02f8eff640 10 monclient: shutdown 2024-03-25T17:57:10.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.258+0000 7f02f8eff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:57:10.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.258+0000 7f02f8eff640 1 -- 172.21.15.18:0/965959589 >> v1:172.21.15.18:6789/0 conn(0x7f02f40b4d60 legacy=0x7f02f413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:10.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.258+0000 7f02f8eff640 1 -- 172.21.15.18:0/965959589 shutdown_connections 2024-03-25T17:57:10.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.258+0000 7f02f8eff640 1 -- 172.21.15.18:0/965959589 >> 172.21.15.18:0/965959589 conn(0x7f02f40617a0 msgr2=0x7f02f4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:57:10.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.258+0000 7f02f8eff640 1 -- 172.21.15.18:0/965959589 shutdown_connections 2024-03-25T17:57:10.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.258+0000 7f02f8eff640 1 -- 172.21.15.18:0/965959589 wait complete. 2024-03-25T17:57:10.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.259+0000 7f02f8eff640 10 monclient: build_initial_monmap 2024-03-25T17:57:10.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.259+0000 7f02f8eff640 10 monclient: monmap: 2024-03-25T17:57:10.261 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:57:10.261 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:57:10.262 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:57:10.262 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:57:10.262 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:57:10.262 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:57:10.262 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:57:10.262 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:57:10.262 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:57:10.262 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:10.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.259+0000 7f02f8eff640 1 librados: starting msgr at 2024-03-25T17:57:10.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.259+0000 7f02f8eff640 1 librados: starting objecter 2024-03-25T17:57:10.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.259+0000 7f02f8eff640 1 Processor -- start 2024-03-25T17:57:10.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.259+0000 7f02f8eff640 1 -- start start 2024-03-25T17:57:10.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.259+0000 7f02f8eff640 1 librados: setting wanted keys 2024-03-25T17:57:10.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.259+0000 7f02f8eff640 1 librados: calling monclient init 2024-03-25T17:57:10.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.259+0000 7f02f8eff640 10 monclient: init 2024-03-25T17:57:10.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.260+0000 7f02f8eff640 10 monclient: _reopen_session rank -1 2024-03-25T17:57:10.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.260+0000 7f02f8eff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:57:10.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.260+0000 7f02f8eff640 10 monclient(hunting): picked mon.noname-c con 0x7f02f40b1730 addr v1:172.21.15.18:6790/0 2024-03-25T17:57:10.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.260+0000 7f02f8eff640 10 monclient(hunting): picked mon.noname-b con 0x7f02f40b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T17:57:10.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.260+0000 7f02f8eff640 10 monclient(hunting): picked mon.noname-a con 0x7f02f414be90 addr v1:172.21.15.18:6789/0 2024-03-25T17:57:10.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.260+0000 7f02f8eff640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f41368f0 con 0x7f02f414be90 2024-03-25T17:57:10.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.260+0000 7f02f8eff640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f4137750 con 0x7f02f40b4d60 2024-03-25T17:57:10.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.260+0000 7f02f8eff640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f41363d0 con 0x7f02f40b1730 2024-03-25T17:57:10.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.260+0000 7f02f8eff640 10 monclient(hunting): _renew_subs 2024-03-25T17:57:10.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.260+0000 7f02f8eff640 10 monclient(hunting): authenticate will time out at 6958.838764s 2024-03-25T17:57:10.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.272+0000 7f02f259c640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f02f40b1730 0x7f02f4135b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:57818/0 (socket says 172.21.15.18:57818) 2024-03-25T17:57:10.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.272+0000 7f02f259c640 1 -- 172.21.15.18:0/4214471517 learned_addr learned my addr 172.21.15.18:0/4214471517 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:57:10.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.273+0000 7f02eaffd640 10 client.?.objecter ms_handle_connect 0x7f02f40b1730 2024-03-25T17:57:10.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.273+0000 7f02eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:10.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.275+0000 7f02eaffd640 10 client.?.objecter ms_handle_connect 0x7f02f414be90 2024-03-25T17:57:10.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.275+0000 7f02eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:10.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.281+0000 7f02eaffd640 10 client.?.objecter ms_handle_connect 0x7f02f40b4d60 2024-03-25T17:57:10.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.281+0000 7f02eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:10.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.283+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034830527 0 0) 0x7f02f41363d0 con 0x7f02f40b1730 2024-03-25T17:57:10.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.283+0000 7f02eaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:57:10.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.283+0000 7f02eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:10.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.283+0000 7f02eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:10.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.283+0000 7f02eaffd640 10 monclient(hunting): my global_id is 4265 2024-03-25T17:57:10.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.283+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d4004eb0 con 0x7f02f40b1730 2024-03-25T17:57:10.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.287+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677054109 0 0) 0x7f02f41368f0 con 0x7f02f414be90 2024-03-25T17:57:10.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.287+0000 7f02eaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:57:10.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.287+0000 7f02eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:10.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.287+0000 7f02eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:10.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.287+0000 7f02eaffd640 10 monclient(hunting): my global_id is 4257 2024-03-25T17:57:10.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.287+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f41363d0 con 0x7f02f414be90 2024-03-25T17:57:10.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.289+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464616647 0 0) 0x7f02f4137750 con 0x7f02f40b4d60 2024-03-25T17:57:10.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.289+0000 7f02eaffd640 10 monclient(hunting): _negotiate 2024-03-25T17:57:10.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.289+0000 7f02eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:10.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.289+0000 7f02eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:10.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.289+0000 7f02eaffd640 10 monclient(hunting): my global_id is 4252 2024-03-25T17:57:10.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.289+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f41368f0 con 0x7f02f40b4d60 2024-03-25T17:57:10.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.303+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 767156656 0 0) 0x7f02f41368f0 con 0x7f02f40b4d60 2024-03-25T17:57:10.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.303+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02f4137750 con 0x7f02f40b4d60 2024-03-25T17:57:10.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.303+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f02dc001860 con 0x7f02f40b4d60 2024-03-25T17:57:10.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.303+0000 7f02eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:10.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.303+0000 7f02eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:57:10.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.303+0000 7f02eaffd640 10 monclient(hunting): dump: 2024-03-25T17:57:10.304 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:10.304 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:10.305 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:10.305 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:10.305 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:10.305 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:10.305 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:10.305 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:10.305 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:10.305 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:10.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.303+0000 7f02eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:57:10.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.311+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1321586241 0 0) 0x7f02d4004eb0 con 0x7f02f40b1730 2024-03-25T17:57:10.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.311+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02f41368f0 con 0x7f02f40b1730 2024-03-25T17:57:10.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.311+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f02e4001690 con 0x7f02f40b1730 2024-03-25T17:57:10.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.311+0000 7f02eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:10.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.311+0000 7f02eaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:57:10.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.311+0000 7f02eaffd640 10 monclient(hunting): dump: 2024-03-25T17:57:10.312 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:10.312 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:10.312 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:10.312 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:10.312 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:10.312 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:10.312 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:10.312 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:10.312 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:10.312 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:10.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.321+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2437153511 0 0) 0x7f02f41363d0 con 0x7f02f414be90 2024-03-25T17:57:10.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.321+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02d4004eb0 con 0x7f02f414be90 2024-03-25T17:57:10.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.321+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f02ec001b90 con 0x7f02f414be90 2024-03-25T17:57:10.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.321+0000 7f02eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:10.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.321+0000 7f02eaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:57:10.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.321+0000 7f02eaffd640 10 monclient(hunting): dump: 2024-03-25T17:57:10.322 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:10.322 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:10.323 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:10.323 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:10.323 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:10.323 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:10.323 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:10.323 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:10.323 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:10.323 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:10.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 130818431 0 0) 0x7f02f41368f0 con 0x7f02f40b1730 2024-03-25T17:57:10.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 >> v1:172.21.15.169:6789/0 conn(0x7f02f40b4d60 legacy=0x7f02f412fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:10.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 >> v1:172.21.15.18:6789/0 conn(0x7f02f414be90 legacy=0x7f02f41d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:10.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02eaffd640 10 monclient: _finish_hunting 0 2024-03-25T17:57:10.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02eaffd640 1 monclient: found mon.c 2024-03-25T17:57:10.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:57:10.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02eaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:10.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f4130470 con 0x7f02f40b1730 2024-03-25T17:57:10.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02eaffd640 10 monclient: _finish_auth 0 2024-03-25T17:57:10.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02eaffd640 10 monclient: _check_auth_tickets 2024-03-25T17:57:10.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:57:10.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02f8eff640 5 monclient: authenticate success, global_id 4265 2024-03-25T17:57:10.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02f8eff640 10 monclient: _renew_subs 2024-03-25T17:57:10.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02f8eff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:10.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02f8eff640 1 -- 172.21.15.18:0/4214471517 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f4130720 con 0x7f02f40b1730 2024-03-25T17:57:10.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02f8eff640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:57:10.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02f8eff640 10 monclient: _renew_subs 2024-03-25T17:57:10.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02f8eff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:10.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02f8eff640 1 -- 172.21.15.18:0/4214471517 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f41322f0 con 0x7f02f40b1730 2024-03-25T17:57:10.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.324+0000 7f02f8eff640 1 librados: init done 2024-03-25T17:57:10.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.325+0000 7f02f8eff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:10.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.325+0000 7f02f8eff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:10.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.325+0000 7f02f8eff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:10.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.325+0000 7f02f8eff640 1 -- 172.21.15.18:0/4214471517 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f4136210 con 0x7f02f40b1730 2024-03-25T17:57:10.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.651+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02e40038a0 con 0x7f02f40b1730 2024-03-25T17:57:10.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.651+0000 7f02eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:57:10.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.651+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f02e4001cc0 con 0x7f02f40b1730 2024-03-25T17:57:10.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.651+0000 7f02eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:10.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.651+0000 7f02eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:57:10.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.651+0000 7f02eaffd640 10 monclient: dump: 2024-03-25T17:57:10.652 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:10.652 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:10.652 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:10.652 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:10.652 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:10.652 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:10.652 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:10.652 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:10.653 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:10.653 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:10.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.651+0000 7f02f159a640 4 set_mon_vals no callback set 2024-03-25T17:57:10.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.652+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f02e4003db0 con 0x7f02f40b1730 2024-03-25T17:57:10.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.652+0000 7f02ebfff640 10 monclient: get_auth_request con 0x7f02d406e160 auth_method 0 2024-03-25T17:57:10.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.652+0000 7f02eaffd640 10 client.4265.objecter ms_handle_connect 0x7f02d406e160 2024-03-25T17:57:10.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.658+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f02e4054950 con 0x7f02f40b1730 2024-03-25T17:57:10.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.658+0000 7f02eaffd640 10 client.4265.objecter ms_dispatch 0x7f02f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:57:10.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.658+0000 7f02eaffd640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:57:10.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.658+0000 7f02eaffd640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:57:10.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.658+0000 7f02eaffd640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-25T17:57:10.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.658+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02e4050460 con 0x7f02f40b1730 2024-03-25T17:57:10.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.658+0000 7f02eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:10.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.658+0000 7f02eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:57:10.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.914+0000 7f02f8eff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:10.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.914+0000 7f02f8eff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:10.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.914+0000 7f02f8eff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:10.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.914+0000 7f02f8eff640 1 -- 172.21.15.18:0/4214471517 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02f40b2270 con 0x7f02f40b1730 2024-03-25T17:57:10.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.942+0000 7f02eaffd640 1 -- 172.21.15.18:0/4214471517 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f02e4052390 con 0x7f02f40b1730 2024-03-25T17:57:10.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.942+0000 7f02eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:10.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.942+0000 7f02eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:57:10.944 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:57:10.944 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:57:10.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.944+0000 7f02f8eff640 10 librados: watch_flush enter 2024-03-25T17:57:10.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.944+0000 7f02f8eff640 10 librados: watch_flush exit 2024-03-25T17:57:10.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.944+0000 7f02f8eff640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-25T17:57:10.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.944+0000 7f02f8eff640 10 client.4265.objecter successfully canceled tick 2024-03-25T17:57:10.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.944+0000 7f02f8eff640 1 -- 172.21.15.18:0/4214471517 >> v1:172.21.15.18:6800/3721244451 conn(0x7f02d406e160 legacy=0x7f02d4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:10.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.944+0000 7f02f8eff640 10 monclient: shutdown 2024-03-25T17:57:10.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.944+0000 7f02f8eff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:57:10.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.944+0000 7f02f8eff640 1 -- 172.21.15.18:0/4214471517 >> v1:172.21.15.18:6790/0 conn(0x7f02f40b1730 legacy=0x7f02f4135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:10.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.944+0000 7f02f8eff640 1 -- 172.21.15.18:0/4214471517 shutdown_connections 2024-03-25T17:57:10.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.944+0000 7f02f8eff640 1 -- 172.21.15.18:0/4214471517 >> 172.21.15.18:0/4214471517 conn(0x7f02f40617a0 msgr2=0x7f02f4138500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:57:10.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.944+0000 7f02f8eff640 1 -- 172.21.15.18:0/4214471517 shutdown_connections 2024-03-25T17:57:10.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.944+0000 7f02f8eff640 1 -- 172.21.15.18:0/4214471517 wait complete. 2024-03-25T17:57:10.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:10.944+0000 7f02f8eff640 1 librados: shutdown 2024-03-25T17:57:10.957 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:57:17.764 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:57:17.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.881+0000 7fca8d231640 10 monclient: get_monmap_and_config 2024-03-25T17:57:17.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.881+0000 7fca8d231640 10 monclient: build_initial_monmap 2024-03-25T17:57:17.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.881+0000 7fca8d231640 10 monclient: monmap: 2024-03-25T17:57:17.882 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:57:17.882 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:17.882 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:57:17.882770+0000 2024-03-25T17:57:17.882 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:57:17.882770+0000 2024-03-25T17:57:17.882 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:57:17.882 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:17.882 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:57:17.882 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:57:17.883 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:57:17.883 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:17.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.882+0000 7fca8d231640 1 Processor -- start 2024-03-25T17:57:17.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.882+0000 7fca8d231640 1 -- start start 2024-03-25T17:57:17.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.882+0000 7fca8d231640 10 monclient: init 2024-03-25T17:57:17.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.882+0000 7fca8d231640 10 monclient: _reopen_session rank -1 2024-03-25T17:57:17.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.882+0000 7fca8d231640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:57:17.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.882+0000 7fca8d231640 10 monclient(hunting): picked mon.noname-c con 0x7fca880a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T17:57:17.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.882+0000 7fca8d231640 10 monclient(hunting): picked mon.noname-b con 0x7fca8813f710 addr v1:172.21.15.18:6790/0 2024-03-25T17:57:17.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.883+0000 7fca8d231640 10 monclient(hunting): picked mon.noname-a con 0x7fca881430c0 addr v1:172.21.15.18:6789/0 2024-03-25T17:57:17.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.883+0000 7fca8d231640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca88142520 con 0x7fca881430c0 2024-03-25T17:57:17.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.883+0000 7fca8d231640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca880b28a0 con 0x7fca880a98f0 2024-03-25T17:57:17.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.883+0000 7fca8d231640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca880b2a30 con 0x7fca8813f710 2024-03-25T17:57:17.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.883+0000 7fca8d231640 10 monclient(hunting): _renew_subs 2024-03-25T17:57:17.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.883+0000 7fca8d231640 10 monclient(hunting): authenticate will time out at 6966.461748s 2024-03-25T17:57:17.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.886+0000 7fca8659c640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fca8813f710 0x7fca881429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:41620/0 (socket says 172.21.15.18:41620) 2024-03-25T17:57:17.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.886+0000 7fca8659c640 1 -- 172.21.15.18:0/1672065413 learned_addr learned my addr 172.21.15.18:0/1672065413 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:57:17.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.897+0000 7fca85d9b640 1 -- 172.21.15.18:0/1672065413 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246156163 0 0) 0x7fca880b2a30 con 0x7fca8813f710 2024-03-25T17:57:17.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.897+0000 7fca85d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:57:17.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.897+0000 7fca85d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:17.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.897+0000 7fca85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:17.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.897+0000 7fca85d9b640 10 monclient(hunting): my global_id is 4268 2024-03-25T17:57:17.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.897+0000 7fca85d9b640 1 -- 172.21.15.18:0/1672065413 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca70003520 con 0x7fca8813f710 2024-03-25T17:57:17.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.905+0000 7fca85d9b640 1 -- 172.21.15.18:0/1672065413 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2026748551 0 0) 0x7fca70003520 con 0x7fca8813f710 2024-03-25T17:57:17.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.905+0000 7fca85d9b640 1 -- 172.21.15.18:0/1672065413 >> v1:172.21.15.169:6789/0 conn(0x7fca880a98f0 legacy=0x7fca880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:17.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.905+0000 7fca85d9b640 1 -- 172.21.15.18:0/1672065413 >> v1:172.21.15.18:6789/0 conn(0x7fca881430c0 legacy=0x7fca8814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:17.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.905+0000 7fca85d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:57:17.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.905+0000 7fca85d9b640 1 monclient: found mon.noname-b 2024-03-25T17:57:17.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.905+0000 7fca85d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:57:17.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.905+0000 7fca85d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T17:57:17.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.905+0000 7fca85d9b640 1 -- 172.21.15.18:0/1672065413 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca880b44c0 con 0x7fca8813f710 2024-03-25T17:57:17.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.905+0000 7fca85d9b640 10 monclient: _finish_auth 0 2024-03-25T17:57:17.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.905+0000 7fca85d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:57:17.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.905+0000 7fca85d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:57:17.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.905+0000 7fca8d231640 5 monclient: authenticate success, global_id 4268 2024-03-25T17:57:17.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.905+0000 7fca85d9b640 1 -- 172.21.15.18:0/1672065413 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fca7c001690 con 0x7fca8813f710 2024-03-25T17:57:17.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.905+0000 7fca8d231640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:57:17.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.906+0000 7fca85d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:17.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.906+0000 7fca85d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:57:17.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.906+0000 7fca85d9b640 10 monclient: dump: 2024-03-25T17:57:17.908 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:17.908 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:17.908 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:17.908 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:17.908 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:17.908 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:17.909 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:17.909 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:17.909 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:17.909 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:17.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.906+0000 7fca8d231640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:57:17.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.928+0000 7fca85d9b640 1 -- 172.21.15.18:0/1672065413 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca7c0017f0 con 0x7fca8813f710 2024-03-25T17:57:17.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.928+0000 7fca85d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:57:17.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.928+0000 7fca85d9b640 1 -- 172.21.15.18:0/1672065413 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fca7c001c00 con 0x7fca8813f710 2024-03-25T17:57:17.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.928+0000 7fca85d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:17.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.928+0000 7fca85d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:57:17.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.928+0000 7fca85d9b640 10 monclient: dump: 2024-03-25T17:57:17.929 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:17.929 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:17.929 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:17.929 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:17.929 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:17.929 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:17.929 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:17.929 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:17.930 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:17.930 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:17.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.928+0000 7fca8d231640 10 monclient: get_monmap_and_config success 2024-03-25T17:57:17.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.928+0000 7fca8d231640 4 set_mon_vals no callback set 2024-03-25T17:57:17.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.929+0000 7fca8d231640 10 monclient: shutdown 2024-03-25T17:57:17.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.929+0000 7fca8d231640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:57:17.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.929+0000 7fca8d231640 1 -- 172.21.15.18:0/1672065413 >> v1:172.21.15.18:6790/0 conn(0x7fca8813f710 legacy=0x7fca881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:17.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.929+0000 7fca8d231640 1 -- 172.21.15.18:0/1672065413 shutdown_connections 2024-03-25T17:57:17.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.929+0000 7fca8d231640 1 -- 172.21.15.18:0/1672065413 >> 172.21.15.18:0/1672065413 conn(0x7fca880b7ea0 msgr2=0x7fca880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:57:17.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.929+0000 7fca8d231640 1 -- 172.21.15.18:0/1672065413 shutdown_connections 2024-03-25T17:57:17.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.929+0000 7fca8d231640 1 -- 172.21.15.18:0/1672065413 wait complete. 2024-03-25T17:57:17.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.930+0000 7fca8d231640 10 monclient: build_initial_monmap 2024-03-25T17:57:17.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.930+0000 7fca8d231640 10 monclient: monmap: 2024-03-25T17:57:17.931 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:57:17.931 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:57:17.932 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:57:17.932 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:57:17.932 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:57:17.932 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:57:17.932 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:57:17.932 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:57:17.932 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:57:17.932 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:17.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.931+0000 7fca8d231640 1 librados: starting msgr at 2024-03-25T17:57:17.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.931+0000 7fca8d231640 1 librados: starting objecter 2024-03-25T17:57:17.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.931+0000 7fca8d231640 1 Processor -- start 2024-03-25T17:57:17.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.931+0000 7fca8d231640 1 -- start start 2024-03-25T17:57:17.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.931+0000 7fca8d231640 1 librados: setting wanted keys 2024-03-25T17:57:17.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.931+0000 7fca8d231640 1 librados: calling monclient init 2024-03-25T17:57:17.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.931+0000 7fca8d231640 10 monclient: init 2024-03-25T17:57:17.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.932+0000 7fca8d231640 10 monclient: _reopen_session rank -1 2024-03-25T17:57:17.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.932+0000 7fca8d231640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:57:17.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.932+0000 7fca8d231640 10 monclient(hunting): picked mon.noname-c con 0x7fca880a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T17:57:17.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.932+0000 7fca8d231640 10 monclient(hunting): picked mon.noname-a con 0x7fca8813f710 addr v1:172.21.15.18:6789/0 2024-03-25T17:57:17.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.932+0000 7fca8d231640 10 monclient(hunting): picked mon.noname-b con 0x7fca881430c0 addr v1:172.21.15.169:6789/0 2024-03-25T17:57:17.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.932+0000 7fca8d231640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca88137c00 con 0x7fca8813f710 2024-03-25T17:57:17.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.932+0000 7fca8d231640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca88139590 con 0x7fca881430c0 2024-03-25T17:57:17.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.932+0000 7fca8d231640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca88139870 con 0x7fca880a98f0 2024-03-25T17:57:17.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.932+0000 7fca8d231640 10 monclient(hunting): _renew_subs 2024-03-25T17:57:17.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.932+0000 7fca8d231640 10 monclient(hunting): authenticate will time out at 6966.511334s 2024-03-25T17:57:17.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.947+0000 7fca86d9d640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fca880a98f0 0x7fca881db170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:41680/0 (socket says 172.21.15.18:41680) 2024-03-25T17:57:17.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.947+0000 7fca86d9d640 1 -- 172.21.15.18:0/1689976152 learned_addr learned my addr 172.21.15.18:0/1689976152 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:57:17.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.949+0000 7fca6f7fe640 10 client.?.objecter ms_handle_connect 0x7fca880a98f0 2024-03-25T17:57:17.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.949+0000 7fca6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:17.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.953+0000 7fca6f7fe640 10 client.?.objecter ms_handle_connect 0x7fca8813f710 2024-03-25T17:57:17.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.953+0000 7fca6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:17.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.954+0000 7fca6f7fe640 10 client.?.objecter ms_handle_connect 0x7fca881430c0 2024-03-25T17:57:17.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.954+0000 7fca6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:17.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.957+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114737325 0 0) 0x7fca88139870 con 0x7fca880a98f0 2024-03-25T17:57:17.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.957+0000 7fca6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:57:17.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.957+0000 7fca6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:17.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.957+0000 7fca6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:17.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.957+0000 7fca6f7fe640 10 monclient(hunting): my global_id is 4271 2024-03-25T17:57:17.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.957+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca74004eb0 con 0x7fca880a98f0 2024-03-25T17:57:17.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.963+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077594876 0 0) 0x7fca88137c00 con 0x7fca8813f710 2024-03-25T17:57:17.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.963+0000 7fca6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:57:17.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.963+0000 7fca6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:17.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.963+0000 7fca6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:17.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.963+0000 7fca6f7fe640 10 monclient(hunting): my global_id is 4263 2024-03-25T17:57:17.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.963+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca88139870 con 0x7fca8813f710 2024-03-25T17:57:17.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.965+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370063902 0 0) 0x7fca88139590 con 0x7fca881430c0 2024-03-25T17:57:17.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.965+0000 7fca6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:57:17.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.965+0000 7fca6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:17.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.965+0000 7fca6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:17.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.965+0000 7fca6f7fe640 10 monclient(hunting): my global_id is 4255 2024-03-25T17:57:17.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.965+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca88137c00 con 0x7fca881430c0 2024-03-25T17:57:17.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.971+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3695311454 0 0) 0x7fca74004eb0 con 0x7fca880a98f0 2024-03-25T17:57:17.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.971+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca88139590 con 0x7fca880a98f0 2024-03-25T17:57:17.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.971+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fca780019d0 con 0x7fca880a98f0 2024-03-25T17:57:17.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.971+0000 7fca6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:17.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.971+0000 7fca6f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:57:17.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.971+0000 7fca6f7fe640 10 monclient(hunting): dump: 2024-03-25T17:57:17.972 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:17.972 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:17.972 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:17.972 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:17.972 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:17.972 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:17.972 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:17.972 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:17.972 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:17.972 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:17.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.971+0000 7fca6f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:57:17.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.973+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 786364274 0 0) 0x7fca88137c00 con 0x7fca881430c0 2024-03-25T17:57:17.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.973+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca74004eb0 con 0x7fca881430c0 2024-03-25T17:57:17.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.973+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fca80001a90 con 0x7fca881430c0 2024-03-25T17:57:17.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.973+0000 7fca6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:17.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.974+0000 7fca6f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:57:17.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.974+0000 7fca6f7fe640 10 monclient(hunting): dump: 2024-03-25T17:57:17.975 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:17.975 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:17.975 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:17.975 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:17.975 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:17.975 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:17.975 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:17.975 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:17.975 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:17.975 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:17.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2916759313 0 0) 0x7fca88139590 con 0x7fca880a98f0 2024-03-25T17:57:17.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 >> v1:172.21.15.169:6789/0 conn(0x7fca881430c0 legacy=0x7fca881e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:17.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 >> v1:172.21.15.18:6789/0 conn(0x7fca8813f710 legacy=0x7fca88134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:17.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca6f7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:57:17.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca6f7fe640 1 monclient: found mon.c 2024-03-25T17:57:17.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:57:17.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca6f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:17.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca881377d0 con 0x7fca880a98f0 2024-03-25T17:57:17.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca6f7fe640 10 monclient: _finish_auth 0 2024-03-25T17:57:17.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca6f7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:57:17.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:57:17.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca8d231640 5 monclient: authenticate success, global_id 4271 2024-03-25T17:57:17.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca8d231640 10 monclient: _renew_subs 2024-03-25T17:57:17.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca8d231640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:17.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca8d231640 1 -- 172.21.15.18:0/1689976152 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca88137a40 con 0x7fca880a98f0 2024-03-25T17:57:17.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca8d231640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:57:17.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca8d231640 10 monclient: _renew_subs 2024-03-25T17:57:17.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca8d231640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:17.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca8d231640 1 -- 172.21.15.18:0/1689976152 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca88134970 con 0x7fca880a98f0 2024-03-25T17:57:17.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.983+0000 7fca8d231640 1 librados: init done 2024-03-25T17:57:17.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.984+0000 7fca8d231640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:17.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.984+0000 7fca8d231640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:17.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.984+0000 7fca8d231640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:17.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:17.984+0000 7fca8d231640 1 -- 172.21.15.18:0/1689976152 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca881dadd0 con 0x7fca880a98f0 2024-03-25T17:57:18.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.294+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca78003610 con 0x7fca880a98f0 2024-03-25T17:57:18.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.294+0000 7fca6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:57:18.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.294+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fca78003800 con 0x7fca880a98f0 2024-03-25T17:57:18.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.294+0000 7fca6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:18.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.294+0000 7fca6f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:57:18.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.294+0000 7fca6f7fe640 10 monclient: dump: 2024-03-25T17:57:18.296 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:18.296 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:18.296 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:18.296 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:18.296 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:18.296 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:18.296 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:18.296 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:18.296 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:18.296 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:18.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.294+0000 7fca8559a640 4 set_mon_vals no callback set 2024-03-25T17:57:18.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.295+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fca78003a20 con 0x7fca880a98f0 2024-03-25T17:57:18.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.296+0000 7fca8659c640 10 monclient: get_auth_request con 0x7fca7406df70 auth_method 0 2024-03-25T17:57:18.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.296+0000 7fca6f7fe640 10 client.4271.objecter ms_handle_connect 0x7fca7406df70 2024-03-25T17:57:18.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.303+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fca78054680 con 0x7fca880a98f0 2024-03-25T17:57:18.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.303+0000 7fca6f7fe640 10 client.4271.objecter ms_dispatch 0x7fca88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:57:18.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.303+0000 7fca6f7fe640 3 client.4271.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:57:18.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.303+0000 7fca6f7fe640 3 client.4271.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:57:18.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.303+0000 7fca6f7fe640 20 client.4271.objecter dump_active .. 0 homeless 2024-03-25T17:57:18.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.303+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca78050190 con 0x7fca880a98f0 2024-03-25T17:57:18.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.303+0000 7fca6f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:18.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.303+0000 7fca6f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:57:18.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.531+0000 7fca8d231640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:18.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.531+0000 7fca8d231640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:18.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.531+0000 7fca8d231640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:18.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.531+0000 7fca8d231640 1 -- 172.21.15.18:0/1689976152 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca88137d90 con 0x7fca880a98f0 2024-03-25T17:57:18.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.554+0000 7fca6f7fe640 1 -- 172.21.15.18:0/1689976152 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fca88137d90 con 0x7fca880a98f0 2024-03-25T17:57:18.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.554+0000 7fca6f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:18.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.554+0000 7fca6f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:57:18.555 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:57:18.555 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:57:18.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.556+0000 7fca8d231640 10 librados: watch_flush enter 2024-03-25T17:57:18.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.556+0000 7fca8d231640 10 librados: watch_flush exit 2024-03-25T17:57:18.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.556+0000 7fca8d231640 20 client.4271.objecter shutdown clearing up homeless session... 2024-03-25T17:57:18.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.556+0000 7fca8d231640 10 client.4271.objecter successfully canceled tick 2024-03-25T17:57:18.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.556+0000 7fca8d231640 1 -- 172.21.15.18:0/1689976152 >> v1:172.21.15.18:6800/3721244451 conn(0x7fca7406df70 legacy=0x7fca74070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:18.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.556+0000 7fca8d231640 10 monclient: shutdown 2024-03-25T17:57:18.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.556+0000 7fca8d231640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:57:18.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.556+0000 7fca8d231640 1 -- 172.21.15.18:0/1689976152 >> v1:172.21.15.18:6790/0 conn(0x7fca880a98f0 legacy=0x7fca881db170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:18.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.556+0000 7fca8d231640 1 -- 172.21.15.18:0/1689976152 shutdown_connections 2024-03-25T17:57:18.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.556+0000 7fca8d231640 1 -- 172.21.15.18:0/1689976152 >> 172.21.15.18:0/1689976152 conn(0x7fca880b7ea0 msgr2=0x7fca88074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:57:18.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.556+0000 7fca8d231640 1 -- 172.21.15.18:0/1689976152 shutdown_connections 2024-03-25T17:57:18.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.556+0000 7fca8d231640 1 -- 172.21.15.18:0/1689976152 wait complete. 2024-03-25T17:57:18.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:18.556+0000 7fca8d231640 1 librados: shutdown 2024-03-25T17:57:18.568 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:57:21.614 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:19.407 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:21.669 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:19.463 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:21.875 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:19.668 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:21.889 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:19.683 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:21.950 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:19.744 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:21.996 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:19.745 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:22.060 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:19.809 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:22.197 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:19.990 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:22.290 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:20.040 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:25.374 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:57:25.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.489+0000 7fabcb40d640 10 monclient: get_monmap_and_config 2024-03-25T17:57:25.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.489+0000 7fabcb40d640 10 monclient: build_initial_monmap 2024-03-25T17:57:25.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.489+0000 7fabcb40d640 10 monclient: monmap: 2024-03-25T17:57:25.490 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:57:25.490 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:25.490 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:57:25.490757+0000 2024-03-25T17:57:25.490 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:57:25.490757+0000 2024-03-25T17:57:25.490 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:57:25.490 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:25.491 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:57:25.491 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:57:25.491 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:57:25.491 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:25.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.490+0000 7fabcb40d640 1 Processor -- start 2024-03-25T17:57:25.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.490+0000 7fabcb40d640 1 -- start start 2024-03-25T17:57:25.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.490+0000 7fabcb40d640 10 monclient: init 2024-03-25T17:57:25.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.491+0000 7fabcb40d640 10 monclient: _reopen_session rank -1 2024-03-25T17:57:25.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.491+0000 7fabcb40d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:57:25.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.491+0000 7fabcb40d640 10 monclient(hunting): picked mon.noname-c con 0x7fabc40a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T17:57:25.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.491+0000 7fabcb40d640 10 monclient(hunting): picked mon.noname-a con 0x7fabc413f710 addr v1:172.21.15.18:6789/0 2024-03-25T17:57:25.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.491+0000 7fabcb40d640 10 monclient(hunting): picked mon.noname-b con 0x7fabc41430c0 addr v1:172.21.15.18:6790/0 2024-03-25T17:57:25.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.491+0000 7fabcb40d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc4142520 con 0x7fabc413f710 2024-03-25T17:57:25.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.491+0000 7fabcb40d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc40b28a0 con 0x7fabc40a98f0 2024-03-25T17:57:25.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.491+0000 7fabcb40d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc40b2a30 con 0x7fabc41430c0 2024-03-25T17:57:25.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.491+0000 7fabcb40d640 10 monclient(hunting): _renew_subs 2024-03-25T17:57:25.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.491+0000 7fabcb40d640 10 monclient(hunting): authenticate will time out at 6974.069930s 2024-03-25T17:57:25.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.499+0000 7fabc91a9640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fabc40a98f0 0x7fabc40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:41020/0 (socket says 172.21.15.18:41020) 2024-03-25T17:57:25.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.499+0000 7fabc91a9640 1 -- 172.21.15.18:0/2233444073 learned_addr learned my addr 172.21.15.18:0/2233444073 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:57:25.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.517+0000 7fabbbfff640 1 -- 172.21.15.18:0/2233444073 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405485690 0 0) 0x7fabc4142520 con 0x7fabc413f710 2024-03-25T17:57:25.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.517+0000 7fabbbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:57:25.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.517+0000 7fabbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:25.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.517+0000 7fabbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:25.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.517+0000 7fabbbfff640 10 monclient(hunting): my global_id is 4266 2024-03-25T17:57:25.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.517+0000 7fabbbfff640 1 -- 172.21.15.18:0/2233444073 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb0003520 con 0x7fabc413f710 2024-03-25T17:57:25.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.521+0000 7fabbbfff640 1 -- 172.21.15.18:0/2233444073 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340177399 0 0) 0x7fabc40b28a0 con 0x7fabc40a98f0 2024-03-25T17:57:25.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.521+0000 7fabbbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:57:25.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.521+0000 7fabbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:25.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.521+0000 7fabbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:25.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.521+0000 7fabbbfff640 10 monclient(hunting): my global_id is 4258 2024-03-25T17:57:25.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.521+0000 7fabbbfff640 1 -- 172.21.15.18:0/2233444073 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc4142520 con 0x7fabc40a98f0 2024-03-25T17:57:25.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.525+0000 7fabbbfff640 1 -- 172.21.15.18:0/2233444073 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967597639 0 0) 0x7fabc40b2a30 con 0x7fabc41430c0 2024-03-25T17:57:25.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.525+0000 7fabbbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:57:25.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.525+0000 7fabbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:25.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.525+0000 7fabbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:25.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.525+0000 7fabbbfff640 10 monclient(hunting): my global_id is 4274 2024-03-25T17:57:25.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.525+0000 7fabbbfff640 1 -- 172.21.15.18:0/2233444073 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc40b28a0 con 0x7fabc41430c0 2024-03-25T17:57:25.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.527+0000 7fabbbfff640 1 -- 172.21.15.18:0/2233444073 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 720754413 0 0) 0x7fabb0003520 con 0x7fabc413f710 2024-03-25T17:57:25.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.527+0000 7fabbbfff640 1 -- 172.21.15.18:0/2233444073 >> v1:172.21.15.18:6790/0 conn(0x7fabc41430c0 legacy=0x7fabc414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:25.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.527+0000 7fabbbfff640 1 -- 172.21.15.18:0/2233444073 >> v1:172.21.15.169:6789/0 conn(0x7fabc40a98f0 legacy=0x7fabc40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:25.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.527+0000 7fabbbfff640 10 monclient: _finish_hunting 0 2024-03-25T17:57:25.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.527+0000 7fabbbfff640 1 monclient: found mon.noname-a 2024-03-25T17:57:25.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.527+0000 7fabbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:57:25.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.527+0000 7fabbbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T17:57:25.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.527+0000 7fabbbfff640 1 -- 172.21.15.18:0/2233444073 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc40b44c0 con 0x7fabc413f710 2024-03-25T17:57:25.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.527+0000 7fabbbfff640 10 monclient: _finish_auth 0 2024-03-25T17:57:25.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.527+0000 7fabbbfff640 10 monclient: _check_auth_tickets 2024-03-25T17:57:25.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.527+0000 7fabbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:57:25.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.527+0000 7fabbbfff640 1 -- 172.21.15.18:0/2233444073 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fabb4001690 con 0x7fabc413f710 2024-03-25T17:57:25.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.527+0000 7fabbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:25.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.527+0000 7fabbbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:57:25.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.527+0000 7fabbbfff640 10 monclient: dump: 2024-03-25T17:57:25.531 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:25.531 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:25.531 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:25.531 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:25.531 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:25.531 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:25.531 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:25.531 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:25.531 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:25.531 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:25.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.527+0000 7fabcb40d640 5 monclient: authenticate success, global_id 4266 2024-03-25T17:57:25.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.527+0000 7fabcb40d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:57:25.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.555+0000 7fabbbfff640 1 -- 172.21.15.18:0/2233444073 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabb40017f0 con 0x7fabc413f710 2024-03-25T17:57:25.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.555+0000 7fabbbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:57:25.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.555+0000 7fabbbfff640 1 -- 172.21.15.18:0/2233444073 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fabb4001c00 con 0x7fabc413f710 2024-03-25T17:57:25.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.555+0000 7fabcb40d640 10 monclient: get_monmap_and_config success 2024-03-25T17:57:25.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.555+0000 7fabcb40d640 4 set_mon_vals no callback set 2024-03-25T17:57:25.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.555+0000 7fabbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:25.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.555+0000 7fabbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:57:25.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.555+0000 7fabbbfff640 10 monclient: dump: 2024-03-25T17:57:25.556 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:25.556 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:25.556 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:25.556 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:25.556 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:25.557 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:25.557 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:25.557 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:25.557 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:25.557 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:25.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.556+0000 7fabcb40d640 10 monclient: shutdown 2024-03-25T17:57:25.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.556+0000 7fabcb40d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:57:25.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.556+0000 7fabcb40d640 1 -- 172.21.15.18:0/2233444073 >> v1:172.21.15.18:6789/0 conn(0x7fabc413f710 legacy=0x7fabc41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:25.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.556+0000 7fabcb40d640 1 -- 172.21.15.18:0/2233444073 shutdown_connections 2024-03-25T17:57:25.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.556+0000 7fabcb40d640 1 -- 172.21.15.18:0/2233444073 >> 172.21.15.18:0/2233444073 conn(0x7fabc40b7ea0 msgr2=0x7fabc40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:57:25.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.556+0000 7fabcb40d640 1 -- 172.21.15.18:0/2233444073 shutdown_connections 2024-03-25T17:57:25.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.556+0000 7fabcb40d640 1 -- 172.21.15.18:0/2233444073 wait complete. 2024-03-25T17:57:25.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.557+0000 7fabcb40d640 10 monclient: build_initial_monmap 2024-03-25T17:57:25.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.557+0000 7fabcb40d640 10 monclient: monmap: 2024-03-25T17:57:25.559 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:57:25.559 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:57:25.559 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:57:25.559 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:57:25.559 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:57:25.559 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:57:25.559 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:57:25.559 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:57:25.560 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:57:25.560 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:25.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.557+0000 7fabcb40d640 1 librados: starting msgr at 2024-03-25T17:57:25.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.557+0000 7fabcb40d640 1 librados: starting objecter 2024-03-25T17:57:25.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.557+0000 7fabcb40d640 1 Processor -- start 2024-03-25T17:57:25.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.558+0000 7fabcb40d640 1 -- start start 2024-03-25T17:57:25.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.558+0000 7fabcb40d640 1 librados: setting wanted keys 2024-03-25T17:57:25.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.558+0000 7fabcb40d640 1 librados: calling monclient init 2024-03-25T17:57:25.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.558+0000 7fabcb40d640 10 monclient: init 2024-03-25T17:57:25.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.558+0000 7fabcb40d640 10 monclient: _reopen_session rank -1 2024-03-25T17:57:25.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.558+0000 7fabcb40d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:57:25.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.558+0000 7fabcb40d640 10 monclient(hunting): picked mon.noname-c con 0x7fabc40a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T17:57:25.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.558+0000 7fabcb40d640 10 monclient(hunting): picked mon.noname-b con 0x7fabc413f710 addr v1:172.21.15.169:6789/0 2024-03-25T17:57:25.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.558+0000 7fabcb40d640 10 monclient(hunting): picked mon.noname-a con 0x7fabc41430c0 addr v1:172.21.15.18:6789/0 2024-03-25T17:57:25.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.558+0000 7fabcb40d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc41d82d0 con 0x7fabc41430c0 2024-03-25T17:57:25.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.558+0000 7fabcb40d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc40abb90 con 0x7fabc413f710 2024-03-25T17:57:25.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.558+0000 7fabcb40d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc41d76a0 con 0x7fabc40a98f0 2024-03-25T17:57:25.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.558+0000 7fabcb40d640 10 monclient(hunting): _renew_subs 2024-03-25T17:57:25.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.558+0000 7fabcb40d640 10 monclient(hunting): authenticate will time out at 6974.137526s 2024-03-25T17:57:25.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.570+0000 7fabc89a8640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fabc413f710 0x7fabc41d7bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:41080/0 (socket says 172.21.15.18:41080) 2024-03-25T17:57:25.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.570+0000 7fabc89a8640 1 -- 172.21.15.18:0/1729320923 learned_addr learned my addr 172.21.15.18:0/1729320923 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:57:25.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.573+0000 7fabb9ffb640 10 client.?.objecter ms_handle_connect 0x7fabc413f710 2024-03-25T17:57:25.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.573+0000 7fabb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:25.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.578+0000 7fabb9ffb640 10 client.?.objecter ms_handle_connect 0x7fabc41430c0 2024-03-25T17:57:25.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.578+0000 7fabb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:25.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.579+0000 7fabb9ffb640 10 client.?.objecter ms_handle_connect 0x7fabc40a98f0 2024-03-25T17:57:25.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.579+0000 7fabb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:25.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.594+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157911710 0 0) 0x7fabc41d82d0 con 0x7fabc41430c0 2024-03-25T17:57:25.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.594+0000 7fabb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:57:25.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.594+0000 7fabb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:25.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.594+0000 7fabb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:25.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.594+0000 7fabb9ffb640 10 monclient(hunting): my global_id is 4269 2024-03-25T17:57:25.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.594+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faba8004eb0 con 0x7fabc41430c0 2024-03-25T17:57:25.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.595+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605628248 0 0) 0x7fabc40abb90 con 0x7fabc413f710 2024-03-25T17:57:25.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.595+0000 7fabb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:57:25.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.595+0000 7fabb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:25.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.595+0000 7fabb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:25.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.595+0000 7fabb9ffb640 10 monclient(hunting): my global_id is 4261 2024-03-25T17:57:25.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.595+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc41d82d0 con 0x7fabc413f710 2024-03-25T17:57:25.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.606+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719467012 0 0) 0x7fabc41d76a0 con 0x7fabc40a98f0 2024-03-25T17:57:25.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.606+0000 7fabb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:57:25.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.606+0000 7fabb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:25.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.606+0000 7fabb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:25.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.606+0000 7fabb9ffb640 10 monclient(hunting): my global_id is 4277 2024-03-25T17:57:25.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.606+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc40abb90 con 0x7fabc40a98f0 2024-03-25T17:57:25.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.609+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2776584479 0 0) 0x7faba8004eb0 con 0x7fabc41430c0 2024-03-25T17:57:25.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.609+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabc41d76a0 con 0x7fabc41430c0 2024-03-25T17:57:25.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.610+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1157406361 0 0) 0x7fabc41d82d0 con 0x7fabc413f710 2024-03-25T17:57:25.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.610+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faba8004eb0 con 0x7fabc413f710 2024-03-25T17:57:25.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.610+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fabb4001b40 con 0x7fabc413f710 2024-03-25T17:57:25.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.610+0000 7fabb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:25.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.610+0000 7fabb9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:57:25.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.610+0000 7fabb9ffb640 10 monclient(hunting): dump: 2024-03-25T17:57:25.611 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:25.611 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:25.611 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:25.611 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:25.611 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:25.611 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:25.611 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:25.612 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:25.612 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:25.612 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:25.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.610+0000 7fabb9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:57:25.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.613+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fabbc001c10 con 0x7fabc41430c0 2024-03-25T17:57:25.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.613+0000 7fabb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:25.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.613+0000 7fabb9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:57:25.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.613+0000 7fabb9ffb640 10 monclient(hunting): dump: 2024-03-25T17:57:25.614 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:25.614 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:25.614 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:25.614 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:25.614 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:25.614 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:25.614 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:25.614 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:25.614 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:25.614 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:25.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.616+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1739528044 0 0) 0x7fabc40abb90 con 0x7fabc40a98f0 2024-03-25T17:57:25.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.616+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabc41d82d0 con 0x7fabc40a98f0 2024-03-25T17:57:25.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.616+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fabc00018f0 con 0x7fabc40a98f0 2024-03-25T17:57:25.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.616+0000 7fabb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:25.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.616+0000 7fabb9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:57:25.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.616+0000 7fabb9ffb640 10 monclient(hunting): dump: 2024-03-25T17:57:25.617 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:25.617 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:25.617 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:25.617 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:25.617 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:25.617 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:25.617 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:25.618 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:25.618 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:25.618 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:25.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.620+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2044563618 0 0) 0x7faba8004eb0 con 0x7fabc413f710 2024-03-25T17:57:25.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 >> v1:172.21.15.18:6790/0 conn(0x7fabc40a98f0 legacy=0x7fabc41385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:25.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 >> v1:172.21.15.18:6789/0 conn(0x7fabc41430c0 legacy=0x7fabc4134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:25.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabb9ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:57:25.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabb9ffb640 1 monclient: found mon.b 2024-03-25T17:57:25.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabb9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:57:25.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabb9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:57:25.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc4134720 con 0x7fabc413f710 2024-03-25T17:57:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabb9ffb640 10 monclient: _finish_auth 0 2024-03-25T17:57:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabb9ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:57:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:57:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabcb40d640 5 monclient: authenticate success, global_id 4261 2024-03-25T17:57:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabcb40d640 10 monclient: _renew_subs 2024-03-25T17:57:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabcb40d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:57:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabcb40d640 1 -- 172.21.15.18:0/1729320923 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc4134970 con 0x7fabc413f710 2024-03-25T17:57:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabcb40d640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:57:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabcb40d640 10 monclient: _renew_subs 2024-03-25T17:57:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabcb40d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:57:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabcb40d640 1 -- 172.21.15.18:0/1729320923 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc4137a10 con 0x7fabc413f710 2024-03-25T17:57:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.621+0000 7fabcb40d640 1 librados: init done 2024-03-25T17:57:25.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.622+0000 7fabcb40d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:25.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.622+0000 7fabcb40d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:25.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.622+0000 7fabcb40d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:57:25.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.622+0000 7fabcb40d640 1 -- 172.21.15.18:0/1729320923 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc4074cc0 con 0x7fabc413f710 2024-03-25T17:57:25.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.893+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabb40011f0 con 0x7fabc413f710 2024-03-25T17:57:25.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.893+0000 7fabb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:57:25.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.893+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fabb4003a40 con 0x7fabc413f710 2024-03-25T17:57:25.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.893+0000 7fabb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:25.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.893+0000 7fabb9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:57:25.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.893+0000 7fabbb7fe640 4 set_mon_vals no callback set 2024-03-25T17:57:25.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.893+0000 7fabb9ffb640 10 monclient: dump: 2024-03-25T17:57:25.895 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:25.895 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:25.895 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:25.895 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:25.895 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:25.895 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:25.895 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:25.895 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:25.895 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:25.895 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:25.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.894+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fabb4003c60 con 0x7fabc413f710 2024-03-25T17:57:25.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.895+0000 7fabc91a9640 10 monclient: get_auth_request con 0x7faba806e070 auth_method 0 2024-03-25T17:57:25.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.895+0000 7fabb9ffb640 10 client.4261.objecter ms_handle_connect 0x7faba806e070 2024-03-25T17:57:25.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.896+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fabb4054880 con 0x7fabc413f710 2024-03-25T17:57:25.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.896+0000 7fabb9ffb640 10 client.4261.objecter ms_dispatch 0x7fabc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:57:25.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.896+0000 7fabb9ffb640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:57:25.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.896+0000 7fabb9ffb640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:57:25.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.896+0000 7fabb9ffb640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-25T17:57:25.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.899+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fabb40501c0 con 0x7fabc413f710 2024-03-25T17:57:25.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.899+0000 7fabb9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:25.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:25.899+0000 7fabb9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:57:26.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.136+0000 7fabcb40d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:26.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.136+0000 7fabcb40d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:26.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.136+0000 7fabcb40d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:57:26.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.136+0000 7fabcb40d640 1 -- 172.21.15.18:0/1729320923 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabc41d7490 con 0x7fabc413f710 2024-03-25T17:57:26.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.192+0000 7fabb9ffb640 1 -- 172.21.15.18:0/1729320923 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fabb40522c0 con 0x7fabc413f710 2024-03-25T17:57:26.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.192+0000 7fabb9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:26.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.192+0000 7fabb9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:57:26.193 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:57:26.193 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:57:26.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.194+0000 7fabcb40d640 10 librados: watch_flush enter 2024-03-25T17:57:26.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.194+0000 7fabcb40d640 10 librados: watch_flush exit 2024-03-25T17:57:26.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.194+0000 7fabcb40d640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-25T17:57:26.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.194+0000 7fabcb40d640 10 client.4261.objecter successfully canceled tick 2024-03-25T17:57:26.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.195+0000 7fabcb40d640 1 -- 172.21.15.18:0/1729320923 >> v1:172.21.15.18:6800/3721244451 conn(0x7faba806e070 legacy=0x7faba8070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:26.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.195+0000 7fabcb40d640 10 monclient: shutdown 2024-03-25T17:57:26.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.195+0000 7fabcb40d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:57:26.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.195+0000 7fabcb40d640 1 -- 172.21.15.18:0/1729320923 >> v1:172.21.15.169:6789/0 conn(0x7fabc413f710 legacy=0x7fabc41d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:26.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.195+0000 7fabcb40d640 1 -- 172.21.15.18:0/1729320923 shutdown_connections 2024-03-25T17:57:26.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.195+0000 7fabcb40d640 1 -- 172.21.15.18:0/1729320923 >> 172.21.15.18:0/1729320923 conn(0x7fabc40b7ea0 msgr2=0x7fabc4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:57:26.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.195+0000 7fabcb40d640 1 -- 172.21.15.18:0/1729320923 shutdown_connections 2024-03-25T17:57:26.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.195+0000 7fabcb40d640 1 -- 172.21.15.18:0/1729320923 wait complete. 2024-03-25T17:57:26.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:26.195+0000 7fabcb40d640 1 librados: shutdown 2024-03-25T17:57:26.207 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:57:33.014 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:57:33.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.112+0000 7ffa593a3640 10 monclient: get_monmap_and_config 2024-03-25T17:57:33.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.112+0000 7ffa593a3640 10 monclient: build_initial_monmap 2024-03-25T17:57:33.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.112+0000 7ffa593a3640 10 monclient: monmap: 2024-03-25T17:57:33.113 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:57:33.113 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:33.113 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:57:33.113856+0000 2024-03-25T17:57:33.113 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:57:33.113856+0000 2024-03-25T17:57:33.113 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:57:33.113 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:33.113 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:57:33.113 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:57:33.113 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:57:33.114 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:33.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.113+0000 7ffa593a3640 1 Processor -- start 2024-03-25T17:57:33.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.113+0000 7ffa593a3640 1 -- start start 2024-03-25T17:57:33.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.113+0000 7ffa593a3640 10 monclient: init 2024-03-25T17:57:33.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.113+0000 7ffa593a3640 10 monclient: _reopen_session rank -1 2024-03-25T17:57:33.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.113+0000 7ffa593a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:57:33.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.113+0000 7ffa593a3640 10 monclient(hunting): picked mon.noname-c con 0x7ffa540b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T17:57:33.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.113+0000 7ffa593a3640 10 monclient(hunting): picked mon.noname-a con 0x7ffa540b1730 addr v1:172.21.15.18:6789/0 2024-03-25T17:57:33.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.113+0000 7ffa593a3640 10 monclient(hunting): picked mon.noname-b con 0x7ffa5414be90 addr v1:172.21.15.18:6790/0 2024-03-25T17:57:33.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.113+0000 7ffa593a3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa540af940 con 0x7ffa540b1730 2024-03-25T17:57:33.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.113+0000 7ffa593a3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa540b0a70 con 0x7ffa540b4d60 2024-03-25T17:57:33.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.113+0000 7ffa593a3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa540b0d50 con 0x7ffa5414be90 2024-03-25T17:57:33.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.113+0000 7ffa593a3640 10 monclient(hunting): _renew_subs 2024-03-25T17:57:33.116 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.113+0000 7ffa593a3640 10 monclient(hunting): authenticate will time out at 6981.692451s 2024-03-25T17:57:33.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.131+0000 7ffa52ffd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7ffa540b4d60 0x7ffa5413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:45616/0 (socket says 172.21.15.18:45616) 2024-03-25T17:57:33.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.131+0000 7ffa52ffd640 1 -- 172.21.15.18:0/3507620799 learned_addr learned my addr 172.21.15.18:0/3507620799 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:57:33.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.145+0000 7ffa51ffb640 1 -- 172.21.15.18:0/3507620799 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078777737 0 0) 0x7ffa540b0d50 con 0x7ffa5414be90 2024-03-25T17:57:33.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.145+0000 7ffa51ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:57:33.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.145+0000 7ffa51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:33.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.145+0000 7ffa51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:33.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.145+0000 7ffa51ffb640 10 monclient(hunting): my global_id is 4280 2024-03-25T17:57:33.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.145+0000 7ffa51ffb640 1 -- 172.21.15.18:0/3507620799 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa3c003520 con 0x7ffa5414be90 2024-03-25T17:57:33.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.147+0000 7ffa51ffb640 1 -- 172.21.15.18:0/3507620799 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691010840 0 0) 0x7ffa540af940 con 0x7ffa540b1730 2024-03-25T17:57:33.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.148+0000 7ffa51ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:57:33.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.148+0000 7ffa51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:33.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.148+0000 7ffa51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:33.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.148+0000 7ffa51ffb640 10 monclient(hunting): my global_id is 4272 2024-03-25T17:57:33.149 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.148+0000 7ffa51ffb640 1 -- 172.21.15.18:0/3507620799 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa540b0d50 con 0x7ffa540b1730 2024-03-25T17:57:33.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.153+0000 7ffa51ffb640 1 -- 172.21.15.18:0/3507620799 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414098147 0 0) 0x7ffa540b0a70 con 0x7ffa540b4d60 2024-03-25T17:57:33.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.153+0000 7ffa51ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:57:33.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.153+0000 7ffa51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:33.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.153+0000 7ffa51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:33.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.153+0000 7ffa51ffb640 10 monclient(hunting): my global_id is 4264 2024-03-25T17:57:33.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.153+0000 7ffa51ffb640 1 -- 172.21.15.18:0/3507620799 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa540af940 con 0x7ffa540b4d60 2024-03-25T17:57:33.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.157+0000 7ffa51ffb640 1 -- 172.21.15.18:0/3507620799 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1664625735 0 0) 0x7ffa3c003520 con 0x7ffa5414be90 2024-03-25T17:57:33.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.157+0000 7ffa51ffb640 1 -- 172.21.15.18:0/3507620799 >> v1:172.21.15.169:6789/0 conn(0x7ffa540b4d60 legacy=0x7ffa5413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:33.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.157+0000 7ffa51ffb640 1 -- 172.21.15.18:0/3507620799 >> v1:172.21.15.18:6789/0 conn(0x7ffa540b1730 legacy=0x7ffa540b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:33.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.157+0000 7ffa51ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:57:33.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.157+0000 7ffa51ffb640 1 monclient: found mon.noname-b 2024-03-25T17:57:33.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.157+0000 7ffa51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:57:33.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.157+0000 7ffa51ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T17:57:33.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.157+0000 7ffa51ffb640 1 -- 172.21.15.18:0/3507620799 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa540b1030 con 0x7ffa5414be90 2024-03-25T17:57:33.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.157+0000 7ffa51ffb640 10 monclient: _finish_auth 0 2024-03-25T17:57:33.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.157+0000 7ffa51ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:57:33.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.157+0000 7ffa51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:57:33.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.157+0000 7ffa51ffb640 1 -- 172.21.15.18:0/3507620799 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ffa4c001a10 con 0x7ffa5414be90 2024-03-25T17:57:33.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.157+0000 7ffa51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:33.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.157+0000 7ffa51ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:57:33.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.157+0000 7ffa51ffb640 10 monclient: dump: 2024-03-25T17:57:33.159 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:33.159 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:33.159 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:33.159 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:33.159 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:33.160 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:33.160 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:33.160 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:33.160 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:33.160 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:33.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.157+0000 7ffa593a3640 5 monclient: authenticate success, global_id 4280 2024-03-25T17:57:33.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.157+0000 7ffa593a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:57:33.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.178+0000 7ffa51ffb640 1 -- 172.21.15.18:0/3507620799 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa4c001b70 con 0x7ffa5414be90 2024-03-25T17:57:33.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.178+0000 7ffa51ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:57:33.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.178+0000 7ffa51ffb640 1 -- 172.21.15.18:0/3507620799 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ffa4c001f80 con 0x7ffa5414be90 2024-03-25T17:57:33.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.178+0000 7ffa51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:33.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.178+0000 7ffa51ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:57:33.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.178+0000 7ffa51ffb640 10 monclient: dump: 2024-03-25T17:57:33.179 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:33.179 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:33.180 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:33.180 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:33.180 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:33.180 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:33.180 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:33.180 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:33.180 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:33.180 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:33.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.178+0000 7ffa593a3640 10 monclient: get_monmap_and_config success 2024-03-25T17:57:33.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.178+0000 7ffa593a3640 4 set_mon_vals no callback set 2024-03-25T17:57:33.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.179+0000 7ffa593a3640 10 monclient: shutdown 2024-03-25T17:57:33.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.179+0000 7ffa593a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:57:33.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.179+0000 7ffa593a3640 1 -- 172.21.15.18:0/3507620799 >> v1:172.21.15.18:6790/0 conn(0x7ffa5414be90 legacy=0x7ffa5414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:33.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.179+0000 7ffa593a3640 1 -- 172.21.15.18:0/3507620799 shutdown_connections 2024-03-25T17:57:33.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.179+0000 7ffa593a3640 1 -- 172.21.15.18:0/3507620799 >> 172.21.15.18:0/3507620799 conn(0x7ffa540617a0 msgr2=0x7ffa54061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:57:33.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.179+0000 7ffa593a3640 1 -- 172.21.15.18:0/3507620799 shutdown_connections 2024-03-25T17:57:33.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.179+0000 7ffa593a3640 1 -- 172.21.15.18:0/3507620799 wait complete. 2024-03-25T17:57:33.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.180+0000 7ffa593a3640 10 monclient: build_initial_monmap 2024-03-25T17:57:33.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.180+0000 7ffa593a3640 10 monclient: monmap: 2024-03-25T17:57:33.183 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:57:33.183 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:57:33.183 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:57:33.184 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:57:33.184 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:57:33.184 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:57:33.184 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:57:33.184 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:57:33.184 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:57:33.184 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:33.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.180+0000 7ffa593a3640 1 librados: starting msgr at 2024-03-25T17:57:33.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.180+0000 7ffa593a3640 1 librados: starting objecter 2024-03-25T17:57:33.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.180+0000 7ffa593a3640 1 Processor -- start 2024-03-25T17:57:33.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.181+0000 7ffa593a3640 1 -- start start 2024-03-25T17:57:33.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.181+0000 7ffa593a3640 1 librados: setting wanted keys 2024-03-25T17:57:33.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.181+0000 7ffa593a3640 1 librados: calling monclient init 2024-03-25T17:57:33.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.181+0000 7ffa593a3640 10 monclient: init 2024-03-25T17:57:33.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.181+0000 7ffa593a3640 10 monclient: _reopen_session rank -1 2024-03-25T17:57:33.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.181+0000 7ffa593a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:57:33.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.181+0000 7ffa593a3640 10 monclient(hunting): picked mon.noname-a con 0x7ffa540b1730 addr v1:172.21.15.18:6789/0 2024-03-25T17:57:33.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.181+0000 7ffa593a3640 10 monclient(hunting): picked mon.noname-b con 0x7ffa540b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T17:57:33.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.181+0000 7ffa593a3640 10 monclient(hunting): picked mon.noname-c con 0x7ffa5414be90 addr v1:172.21.15.18:6790/0 2024-03-25T17:57:33.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.181+0000 7ffa593a3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa54137b10 con 0x7ffa540b1730 2024-03-25T17:57:33.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.181+0000 7ffa593a3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa541377e0 con 0x7ffa540b4d60 2024-03-25T17:57:33.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.181+0000 7ffa593a3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa54137fa0 con 0x7ffa5414be90 2024-03-25T17:57:33.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.181+0000 7ffa593a3640 10 monclient(hunting): _renew_subs 2024-03-25T17:57:33.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.181+0000 7ffa593a3640 10 monclient(hunting): authenticate will time out at 6981.760347s 2024-03-25T17:57:33.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.190+0000 7ffa537fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7ffa5414be90 0x7ffa541e3270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:53648/0 (socket says 172.21.15.18:53648) 2024-03-25T17:57:33.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.190+0000 7ffa537fe640 1 -- 172.21.15.18:0/881202019 learned_addr learned my addr 172.21.15.18:0/881202019 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:57:33.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.192+0000 7ffa3b7fe640 10 client.?.objecter ms_handle_connect 0x7ffa5414be90 2024-03-25T17:57:33.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.192+0000 7ffa3b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:33.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.192+0000 7ffa3b7fe640 10 client.?.objecter ms_handle_connect 0x7ffa540b4d60 2024-03-25T17:57:33.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.192+0000 7ffa3b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:33.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.201+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450455082 0 0) 0x7ffa541377e0 con 0x7ffa540b4d60 2024-03-25T17:57:33.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.201+0000 7ffa3b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:57:33.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.201+0000 7ffa3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:33.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.201+0000 7ffa3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:33.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.201+0000 7ffa3b7fe640 10 monclient(hunting): my global_id is 4267 2024-03-25T17:57:33.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.201+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa40004eb0 con 0x7ffa540b4d60 2024-03-25T17:57:33.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.203+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210594031 0 0) 0x7ffa54137fa0 con 0x7ffa5414be90 2024-03-25T17:57:33.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.203+0000 7ffa3b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:57:33.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.203+0000 7ffa3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:33.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.203+0000 7ffa3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:33.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.203+0000 7ffa3b7fe640 10 monclient(hunting): my global_id is 4283 2024-03-25T17:57:33.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.203+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa541377e0 con 0x7ffa5414be90 2024-03-25T17:57:33.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.207+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 847438177 0 0) 0x7ffa40004eb0 con 0x7ffa540b4d60 2024-03-25T17:57:33.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.207+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa54137fa0 con 0x7ffa540b4d60 2024-03-25T17:57:33.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.208+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ffa48001a30 con 0x7ffa540b4d60 2024-03-25T17:57:33.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.208+0000 7ffa3b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:33.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.208+0000 7ffa3b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:57:33.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.208+0000 7ffa3b7fe640 10 monclient(hunting): dump: 2024-03-25T17:57:33.209 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:33.209 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:33.209 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:33.209 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:33.209 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:33.209 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:33.209 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:33.209 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:33.209 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:33.209 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:33.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.208+0000 7ffa3b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:57:33.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.210+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3352225969 0 0) 0x7ffa541377e0 con 0x7ffa5414be90 2024-03-25T17:57:33.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.210+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa40004eb0 con 0x7ffa5414be90 2024-03-25T17:57:33.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.210+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ffa4c001ec0 con 0x7ffa5414be90 2024-03-25T17:57:33.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.210+0000 7ffa3b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:33.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.210+0000 7ffa3b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:57:33.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.210+0000 7ffa3b7fe640 10 monclient(hunting): dump: 2024-03-25T17:57:33.211 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:33.211 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:33.211 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:33.212 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:33.212 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:33.212 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:33.212 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:33.212 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:33.212 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:33.212 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:33.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.213+0000 7ffa3b7fe640 10 client.?.objecter ms_handle_connect 0x7ffa540b1730 2024-03-25T17:57:33.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.213+0000 7ffa3b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:33.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1336407368 0 0) 0x7ffa40004eb0 con 0x7ffa5414be90 2024-03-25T17:57:33.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 >> v1:172.21.15.169:6789/0 conn(0x7ffa540b4d60 legacy=0x7ffa541daed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:33.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 >> v1:172.21.15.18:6789/0 conn(0x7ffa540b1730 legacy=0x7ffa54134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:33.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa3b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:57:33.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa3b7fe640 1 monclient: found mon.c 2024-03-25T17:57:33.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:57:33.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa3b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:33.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa54142030 con 0x7ffa5414be90 2024-03-25T17:57:33.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa3b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:57:33.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa3b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:57:33.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:57:33.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa593a3640 5 monclient: authenticate success, global_id 4283 2024-03-25T17:57:33.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa593a3640 10 monclient: _renew_subs 2024-03-25T17:57:33.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa593a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:33.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa593a3640 1 -- 172.21.15.18:0/881202019 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa54140840 con 0x7ffa5414be90 2024-03-25T17:57:33.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa593a3640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:57:33.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa593a3640 10 monclient: _renew_subs 2024-03-25T17:57:33.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa593a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:33.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa593a3640 1 -- 172.21.15.18:0/881202019 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa541357a0 con 0x7ffa5414be90 2024-03-25T17:57:33.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.223+0000 7ffa593a3640 1 librados: init done 2024-03-25T17:57:33.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.224+0000 7ffa593a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:33.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.224+0000 7ffa593a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:33.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.224+0000 7ffa593a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:33.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.224+0000 7ffa593a3640 1 -- 172.21.15.18:0/881202019 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa540627b0 con 0x7ffa5414be90 2024-03-25T17:57:33.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.547+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa4c0020c0 con 0x7ffa5414be90 2024-03-25T17:57:33.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.547+0000 7ffa3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:57:33.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.547+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ffa4c0015d0 con 0x7ffa5414be90 2024-03-25T17:57:33.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.547+0000 7ffa3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:33.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.547+0000 7ffa3b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:57:33.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.547+0000 7ffa3b7fe640 10 monclient: dump: 2024-03-25T17:57:33.548 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:33.548 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:33.548 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:33.549 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:33.549 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:33.549 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:33.549 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:33.549 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:33.549 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:33.549 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:33.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.547+0000 7ffa517fa640 4 set_mon_vals no callback set 2024-03-25T17:57:33.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.548+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7ffa4c003e80 con 0x7ffa5414be90 2024-03-25T17:57:33.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.548+0000 7ffa52ffd640 10 monclient: get_auth_request con 0x7ffa4006de80 auth_method 0 2024-03-25T17:57:33.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.549+0000 7ffa3b7fe640 10 client.4283.objecter ms_handle_connect 0x7ffa4006de80 2024-03-25T17:57:33.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.555+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7ffa4c0549f0 con 0x7ffa5414be90 2024-03-25T17:57:33.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.555+0000 7ffa3b7fe640 10 client.4283.objecter ms_dispatch 0x7ffa54000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:57:33.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.555+0000 7ffa3b7fe640 3 client.4283.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:57:33.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.555+0000 7ffa3b7fe640 3 client.4283.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:57:33.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.555+0000 7ffa3b7fe640 20 client.4283.objecter dump_active .. 0 homeless 2024-03-25T17:57:33.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.555+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa4c050500 con 0x7ffa5414be90 2024-03-25T17:57:33.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.555+0000 7ffa3b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:33.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.555+0000 7ffa3b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:57:33.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.819+0000 7ffa593a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:33.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.819+0000 7ffa593a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:33.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.819+0000 7ffa593a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:57:33.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.819+0000 7ffa593a3640 1 -- 172.21.15.18:0/881202019 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa54137cc0 con 0x7ffa5414be90 2024-03-25T17:57:33.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.853+0000 7ffa3b7fe640 1 -- 172.21.15.18:0/881202019 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7ffa4c052430 con 0x7ffa5414be90 2024-03-25T17:57:33.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.853+0000 7ffa3b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:33.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.853+0000 7ffa3b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:57:33.855 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:57:33.855 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:57:33.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.856+0000 7ffa593a3640 10 librados: watch_flush enter 2024-03-25T17:57:33.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.856+0000 7ffa593a3640 10 librados: watch_flush exit 2024-03-25T17:57:33.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.856+0000 7ffa593a3640 20 client.4283.objecter shutdown clearing up homeless session... 2024-03-25T17:57:33.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.856+0000 7ffa593a3640 10 client.4283.objecter successfully canceled tick 2024-03-25T17:57:33.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.856+0000 7ffa593a3640 1 -- 172.21.15.18:0/881202019 >> v1:172.21.15.18:6800/3721244451 conn(0x7ffa4006de80 legacy=0x7ffa40070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:33.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.856+0000 7ffa593a3640 10 monclient: shutdown 2024-03-25T17:57:33.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.856+0000 7ffa593a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:57:33.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.856+0000 7ffa593a3640 1 -- 172.21.15.18:0/881202019 >> v1:172.21.15.18:6790/0 conn(0x7ffa5414be90 legacy=0x7ffa541e3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:33.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.856+0000 7ffa593a3640 1 -- 172.21.15.18:0/881202019 shutdown_connections 2024-03-25T17:57:33.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.856+0000 7ffa593a3640 1 -- 172.21.15.18:0/881202019 >> 172.21.15.18:0/881202019 conn(0x7ffa540617a0 msgr2=0x7ffa5413c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:57:33.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.856+0000 7ffa593a3640 1 -- 172.21.15.18:0/881202019 shutdown_connections 2024-03-25T17:57:33.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.856+0000 7ffa593a3640 1 -- 172.21.15.18:0/881202019 wait complete. 2024-03-25T17:57:33.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:33.856+0000 7ffa593a3640 1 librados: shutdown 2024-03-25T17:57:33.869 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:57:40.676 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:57:40.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.791+0000 7fac8470e640 10 monclient: get_monmap_and_config 2024-03-25T17:57:40.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.791+0000 7fac8470e640 10 monclient: build_initial_monmap 2024-03-25T17:57:40.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.791+0000 7fac8470e640 10 monclient: monmap: 2024-03-25T17:57:40.792 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:57:40.792 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:40.792 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:57:40.792859+0000 2024-03-25T17:57:40.792 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:57:40.792859+0000 2024-03-25T17:57:40.792 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:57:40.792 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:40.793 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:57:40.793 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:57:40.793 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:57:40.793 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:40.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.792+0000 7fac8470e640 1 Processor -- start 2024-03-25T17:57:40.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.792+0000 7fac8470e640 1 -- start start 2024-03-25T17:57:40.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.792+0000 7fac8470e640 10 monclient: init 2024-03-25T17:57:40.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.793+0000 7fac8470e640 10 monclient: _reopen_session rank -1 2024-03-25T17:57:40.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.793+0000 7fac8470e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:57:40.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.793+0000 7fac8470e640 10 monclient(hunting): picked mon.noname-c con 0x7fac7c0b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T17:57:40.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.793+0000 7fac8470e640 10 monclient(hunting): picked mon.noname-a con 0x7fac7c0b1730 addr v1:172.21.15.18:6789/0 2024-03-25T17:57:40.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.793+0000 7fac8470e640 10 monclient(hunting): picked mon.noname-b con 0x7fac7c14be90 addr v1:172.21.15.18:6790/0 2024-03-25T17:57:40.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.793+0000 7fac8470e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7c0af940 con 0x7fac7c0b1730 2024-03-25T17:57:40.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.793+0000 7fac8470e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7c0b0a70 con 0x7fac7c0b4d60 2024-03-25T17:57:40.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.793+0000 7fac8470e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7c0b0d50 con 0x7fac7c14be90 2024-03-25T17:57:40.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.793+0000 7fac8470e640 10 monclient(hunting): _renew_subs 2024-03-25T17:57:40.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.793+0000 7fac8470e640 10 monclient(hunting): authenticate will time out at 6989.372040s 2024-03-25T17:57:40.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.803+0000 7fac824aa640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fac7c0b4d60 0x7fac7c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:37400/0 (socket says 172.21.15.18:37400) 2024-03-25T17:57:40.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.803+0000 7fac824aa640 1 -- 172.21.15.18:0/3923020298 learned_addr learned my addr 172.21.15.18:0/3923020298 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:57:40.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.823+0000 7fac814a8640 1 -- 172.21.15.18:0/3923020298 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952906632 0 0) 0x7fac7c0b0a70 con 0x7fac7c0b4d60 2024-03-25T17:57:40.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.823+0000 7fac814a8640 10 monclient(hunting): _negotiate 2024-03-25T17:57:40.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.823+0000 7fac814a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:40.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.823+0000 7fac814a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:40.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.823+0000 7fac814a8640 10 monclient(hunting): my global_id is 4270 2024-03-25T17:57:40.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.823+0000 7fac814a8640 1 -- 172.21.15.18:0/3923020298 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac68003520 con 0x7fac7c0b4d60 2024-03-25T17:57:40.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac814a8640 1 -- 172.21.15.18:0/3923020298 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3289881357 0 0) 0x7fac68003520 con 0x7fac7c0b4d60 2024-03-25T17:57:40.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac814a8640 1 -- 172.21.15.18:0/3923020298 >> v1:172.21.15.18:6790/0 conn(0x7fac7c14be90 legacy=0x7fac7c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:40.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac814a8640 1 -- 172.21.15.18:0/3923020298 >> v1:172.21.15.18:6789/0 conn(0x7fac7c0b1730 legacy=0x7fac7c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:40.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac814a8640 10 monclient: _finish_hunting 0 2024-03-25T17:57:40.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac814a8640 1 monclient: found mon.noname-c 2024-03-25T17:57:40.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac814a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:57:40.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac814a8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T17:57:40.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac814a8640 1 -- 172.21.15.18:0/3923020298 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7c0b1030 con 0x7fac7c0b4d60 2024-03-25T17:57:40.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac814a8640 10 monclient: _finish_auth 0 2024-03-25T17:57:40.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac814a8640 10 monclient: _check_auth_tickets 2024-03-25T17:57:40.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac814a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:57:40.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac814a8640 1 -- 172.21.15.18:0/3923020298 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fac78001690 con 0x7fac7c0b4d60 2024-03-25T17:57:40.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac8470e640 5 monclient: authenticate success, global_id 4270 2024-03-25T17:57:40.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac8470e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:57:40.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac814a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:40.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac814a8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:57:40.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac814a8640 10 monclient: dump: 2024-03-25T17:57:40.834 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:40.834 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:40.834 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:40.834 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:40.834 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:40.834 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:40.834 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:40.835 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:40.835 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:40.835 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:40.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.830+0000 7fac8470e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:57:40.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.845+0000 7fac814a8640 1 -- 172.21.15.18:0/3923020298 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac780017f0 con 0x7fac7c0b4d60 2024-03-25T17:57:40.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.845+0000 7fac814a8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:57:40.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.845+0000 7fac814a8640 1 -- 172.21.15.18:0/3923020298 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fac78001c00 con 0x7fac7c0b4d60 2024-03-25T17:57:40.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.845+0000 7fac814a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:40.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.845+0000 7fac814a8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:57:40.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.845+0000 7fac814a8640 10 monclient: dump: 2024-03-25T17:57:40.846 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:40.846 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:40.846 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:40.846 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:40.847 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:40.847 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:40.847 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:40.848 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:40.848 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:40.848 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:40.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.845+0000 7fac8470e640 10 monclient: get_monmap_and_config success 2024-03-25T17:57:40.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.845+0000 7fac8470e640 4 set_mon_vals no callback set 2024-03-25T17:57:40.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.846+0000 7fac8470e640 10 monclient: shutdown 2024-03-25T17:57:40.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.846+0000 7fac8470e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:57:40.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.846+0000 7fac8470e640 1 -- 172.21.15.18:0/3923020298 >> v1:172.21.15.169:6789/0 conn(0x7fac7c0b4d60 legacy=0x7fac7c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:40.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.846+0000 7fac8470e640 1 -- 172.21.15.18:0/3923020298 shutdown_connections 2024-03-25T17:57:40.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.846+0000 7fac8470e640 1 -- 172.21.15.18:0/3923020298 >> 172.21.15.18:0/3923020298 conn(0x7fac7c0617a0 msgr2=0x7fac7c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:57:40.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.846+0000 7fac8470e640 1 -- 172.21.15.18:0/3923020298 shutdown_connections 2024-03-25T17:57:40.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.846+0000 7fac8470e640 1 -- 172.21.15.18:0/3923020298 wait complete. 2024-03-25T17:57:40.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.847+0000 7fac8470e640 10 monclient: build_initial_monmap 2024-03-25T17:57:40.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.847+0000 7fac8470e640 10 monclient: monmap: 2024-03-25T17:57:40.851 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:57:40.851 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:57:40.851 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:57:40.851 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:57:40.852 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:57:40.852 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:57:40.852 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:57:40.852 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:57:40.852 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:57:40.852 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:40.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.847+0000 7fac8470e640 1 librados: starting msgr at 2024-03-25T17:57:40.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.847+0000 7fac8470e640 1 librados: starting objecter 2024-03-25T17:57:40.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.847+0000 7fac8470e640 1 Processor -- start 2024-03-25T17:57:40.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.847+0000 7fac8470e640 1 -- start start 2024-03-25T17:57:40.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.847+0000 7fac8470e640 1 librados: setting wanted keys 2024-03-25T17:57:40.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.847+0000 7fac8470e640 1 librados: calling monclient init 2024-03-25T17:57:40.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.847+0000 7fac8470e640 10 monclient: init 2024-03-25T17:57:40.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.848+0000 7fac8470e640 10 monclient: _reopen_session rank -1 2024-03-25T17:57:40.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.848+0000 7fac8470e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:57:40.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.848+0000 7fac8470e640 10 monclient(hunting): picked mon.noname-a con 0x7fac7c0b1730 addr v1:172.21.15.18:6789/0 2024-03-25T17:57:40.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.848+0000 7fac8470e640 10 monclient(hunting): picked mon.noname-b con 0x7fac7c0b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T17:57:40.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.848+0000 7fac8470e640 10 monclient(hunting): picked mon.noname-c con 0x7fac7c14be90 addr v1:172.21.15.18:6790/0 2024-03-25T17:57:40.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.848+0000 7fac8470e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7c142050 con 0x7fac7c0b1730 2024-03-25T17:57:40.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.848+0000 7fac8470e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7c1dac50 con 0x7fac7c0b4d60 2024-03-25T17:57:40.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.848+0000 7fac8470e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7c140d00 con 0x7fac7c14be90 2024-03-25T17:57:40.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.848+0000 7fac8470e640 10 monclient(hunting): _renew_subs 2024-03-25T17:57:40.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.848+0000 7fac8470e640 10 monclient(hunting): authenticate will time out at 6989.426940s 2024-03-25T17:57:40.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.860+0000 7fac824aa640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fac7c0b1730 0x7fac7c1405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:49702/0 (socket says 172.21.15.18:49702) 2024-03-25T17:57:40.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.860+0000 7fac824aa640 1 -- 172.21.15.18:0/1962511073 learned_addr learned my addr 172.21.15.18:0/1962511073 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:57:40.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.862+0000 7fac72ffd640 10 client.?.objecter ms_handle_connect 0x7fac7c0b1730 2024-03-25T17:57:40.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.862+0000 7fac72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:40.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.863+0000 7fac72ffd640 10 client.?.objecter ms_handle_connect 0x7fac7c14be90 2024-03-25T17:57:40.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.863+0000 7fac72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:40.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.877+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531813734 0 0) 0x7fac7c140d00 con 0x7fac7c14be90 2024-03-25T17:57:40.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.877+0000 7fac72ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:57:40.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.877+0000 7fac72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:40.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.877+0000 7fac72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:40.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.877+0000 7fac72ffd640 10 monclient(hunting): my global_id is 4289 2024-03-25T17:57:40.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.877+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac60004eb0 con 0x7fac7c14be90 2024-03-25T17:57:40.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.877+0000 7fac72ffd640 10 client.?.objecter ms_handle_connect 0x7fac7c0b4d60 2024-03-25T17:57:40.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.877+0000 7fac72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:40.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.886+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989098156 0 0) 0x7fac7c142050 con 0x7fac7c0b1730 2024-03-25T17:57:40.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.886+0000 7fac72ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:57:40.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.886+0000 7fac72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:40.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.886+0000 7fac72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:40.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.886+0000 7fac72ffd640 10 monclient(hunting): my global_id is 4281 2024-03-25T17:57:40.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.886+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac7c140d00 con 0x7fac7c0b1730 2024-03-25T17:57:40.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.886+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874834308 0 0) 0x7fac7c1dac50 con 0x7fac7c0b4d60 2024-03-25T17:57:40.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.886+0000 7fac72ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:57:40.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.886+0000 7fac72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:40.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.886+0000 7fac72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:40.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.886+0000 7fac72ffd640 10 monclient(hunting): my global_id is 4273 2024-03-25T17:57:40.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.886+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac7c142050 con 0x7fac7c0b4d60 2024-03-25T17:57:40.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.896+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 952884430 0 0) 0x7fac7c142050 con 0x7fac7c0b4d60 2024-03-25T17:57:40.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.896+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac7c1dac50 con 0x7fac7c0b4d60 2024-03-25T17:57:40.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.896+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fac6c001860 con 0x7fac7c0b4d60 2024-03-25T17:57:40.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.896+0000 7fac72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:40.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.896+0000 7fac72ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:57:40.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.896+0000 7fac72ffd640 10 monclient(hunting): dump: 2024-03-25T17:57:40.898 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:40.898 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:40.898 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:40.899 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:40.899 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:40.899 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:40.899 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:40.899 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:40.900 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:40.900 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:40.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.896+0000 7fac72ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:57:40.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.900+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2643040995 0 0) 0x7fac7c140d00 con 0x7fac7c0b1730 2024-03-25T17:57:40.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.900+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac7c142050 con 0x7fac7c0b1730 2024-03-25T17:57:40.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.900+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fac78001bc0 con 0x7fac7c0b1730 2024-03-25T17:57:40.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.900+0000 7fac72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:40.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.900+0000 7fac72ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:57:40.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.900+0000 7fac72ffd640 10 monclient(hunting): dump: 2024-03-25T17:57:40.902 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:40.902 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:40.902 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:40.903 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:40.903 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:40.903 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:40.903 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:40.903 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:40.903 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:40.903 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:40.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.902+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 442909110 0 0) 0x7fac60004eb0 con 0x7fac7c14be90 2024-03-25T17:57:40.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.902+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac7c140d00 con 0x7fac7c14be90 2024-03-25T17:57:40.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.902+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fac74001b90 con 0x7fac7c14be90 2024-03-25T17:57:40.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.902+0000 7fac72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:40.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.902+0000 7fac72ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:57:40.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.902+0000 7fac72ffd640 10 monclient(hunting): dump: 2024-03-25T17:57:40.904 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:40.904 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:40.904 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:40.904 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:40.904 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:40.905 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:40.905 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:40.905 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:40.905 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:40.905 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:40.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.909+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3183849752 0 0) 0x7fac7c1dac50 con 0x7fac7c0b4d60 2024-03-25T17:57:40.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 >> v1:172.21.15.18:6790/0 conn(0x7fac7c14be90 legacy=0x7fac7c1e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:40.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 >> v1:172.21.15.18:6789/0 conn(0x7fac7c0b1730 legacy=0x7fac7c1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:40.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac72ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:57:40.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac72ffd640 1 monclient: found mon.b 2024-03-25T17:57:40.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:57:40.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac72ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:57:40.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7c1423e0 con 0x7fac7c0b4d60 2024-03-25T17:57:40.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac72ffd640 10 monclient: _finish_auth 0 2024-03-25T17:57:40.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac72ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:57:40.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:57:40.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac8470e640 5 monclient: authenticate success, global_id 4273 2024-03-25T17:57:40.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac8470e640 10 monclient: _renew_subs 2024-03-25T17:57:40.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac8470e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:57:40.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac8470e640 1 -- 172.21.15.18:0/1962511073 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac7c13b470 con 0x7fac7c0b4d60 2024-03-25T17:57:40.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac8470e640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:57:40.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac8470e640 10 monclient: _renew_subs 2024-03-25T17:57:40.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac8470e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:57:40.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac8470e640 1 -- 172.21.15.18:0/1962511073 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac7c1dba40 con 0x7fac7c0b4d60 2024-03-25T17:57:40.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.910+0000 7fac8470e640 1 librados: init done 2024-03-25T17:57:40.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.911+0000 7fac8470e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:40.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.911+0000 7fac8470e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:40.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.911+0000 7fac8470e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:57:40.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:40.911+0000 7fac8470e640 1 -- 172.21.15.18:0/1962511073 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac7c1401f0 con 0x7fac7c0b4d60 2024-03-25T17:57:41.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.197+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac6c001d30 con 0x7fac7c0b4d60 2024-03-25T17:57:41.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.197+0000 7fac72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:57:41.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.197+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fac6c003790 con 0x7fac7c0b4d60 2024-03-25T17:57:41.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.197+0000 7fac72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:41.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.197+0000 7fac72ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:57:41.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.197+0000 7fac72ffd640 10 monclient: dump: 2024-03-25T17:57:41.198 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:41.198 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:41.199 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:41.199 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:41.199 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:41.199 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:41.199 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:41.199 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:41.199 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:41.200 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:41.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.197+0000 7fac80ca7640 4 set_mon_vals no callback set 2024-03-25T17:57:41.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.198+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fac6c0197e0 con 0x7fac7c0b4d60 2024-03-25T17:57:41.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.199+0000 7fac824aa640 10 monclient: get_auth_request con 0x7fac6006e0f0 auth_method 0 2024-03-25T17:57:41.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.199+0000 7fac72ffd640 10 client.4273.objecter ms_handle_connect 0x7fac6006e0f0 2024-03-25T17:57:41.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.218+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fac6c0534f0 con 0x7fac7c0b4d60 2024-03-25T17:57:41.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.218+0000 7fac72ffd640 10 client.4273.objecter ms_dispatch 0x7fac7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:57:41.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.218+0000 7fac72ffd640 3 client.4273.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:57:41.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.218+0000 7fac72ffd640 3 client.4273.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:57:41.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.218+0000 7fac72ffd640 20 client.4273.objecter dump_active .. 0 homeless 2024-03-25T17:57:41.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.219+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac6c04ee30 con 0x7fac7c0b4d60 2024-03-25T17:57:41.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.219+0000 7fac72ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:41.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.219+0000 7fac72ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:57:41.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.440+0000 7fac8470e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:41.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.440+0000 7fac8470e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:41.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.440+0000 7fac8470e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:57:41.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.440+0000 7fac8470e640 1 -- 172.21.15.18:0/1962511073 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac7c13b600 con 0x7fac7c0b4d60 2024-03-25T17:57:41.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.612+0000 7fac72ffd640 1 -- 172.21.15.18:0/1962511073 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fac6c050f30 con 0x7fac7c0b4d60 2024-03-25T17:57:41.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.613+0000 7fac72ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:41.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.613+0000 7fac72ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:57:41.614 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:57:41.614 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:57:41.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.615+0000 7fac8470e640 10 librados: watch_flush enter 2024-03-25T17:57:41.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.615+0000 7fac8470e640 10 librados: watch_flush exit 2024-03-25T17:57:41.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.615+0000 7fac8470e640 20 client.4273.objecter shutdown clearing up homeless session... 2024-03-25T17:57:41.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.615+0000 7fac8470e640 10 client.4273.objecter successfully canceled tick 2024-03-25T17:57:41.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.615+0000 7fac8470e640 1 -- 172.21.15.18:0/1962511073 >> v1:172.21.15.18:6800/3721244451 conn(0x7fac6006e0f0 legacy=0x7fac600705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:41.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.615+0000 7fac8470e640 10 monclient: shutdown 2024-03-25T17:57:41.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.615+0000 7fac8470e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:57:41.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.615+0000 7fac8470e640 1 -- 172.21.15.18:0/1962511073 >> v1:172.21.15.169:6789/0 conn(0x7fac7c0b4d60 legacy=0x7fac7c1db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:41.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.615+0000 7fac8470e640 1 -- 172.21.15.18:0/1962511073 shutdown_connections 2024-03-25T17:57:41.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.615+0000 7fac8470e640 1 -- 172.21.15.18:0/1962511073 >> 172.21.15.18:0/1962511073 conn(0x7fac7c0617a0 msgr2=0x7fac7c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:57:41.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.615+0000 7fac8470e640 1 -- 172.21.15.18:0/1962511073 shutdown_connections 2024-03-25T17:57:41.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.615+0000 7fac8470e640 1 -- 172.21.15.18:0/1962511073 wait complete. 2024-03-25T17:57:41.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:41.615+0000 7fac8470e640 1 librados: shutdown 2024-03-25T17:57:41.627 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:57:41.631 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:39.425 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:41.685 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:39.478 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:41.729 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:39.478 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:41.737 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:39.487 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:41.758 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:39.551 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:41.788 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:39.537 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:41.788 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:39.581 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:41.794 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:39.587 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:41.847 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:39.640 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:57:48.434 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:57:48.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.546+0000 7f34358f1640 10 monclient: get_monmap_and_config 2024-03-25T17:57:48.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.546+0000 7f34358f1640 10 monclient: build_initial_monmap 2024-03-25T17:57:48.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.546+0000 7f34358f1640 10 monclient: monmap: 2024-03-25T17:57:48.547 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:57:48.547 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:48.548 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:57:48.547919+0000 2024-03-25T17:57:48.548 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:57:48.547919+0000 2024-03-25T17:57:48.548 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:57:48.548 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:48.548 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:57:48.548 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:57:48.548 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:57:48.548 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:48.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.547+0000 7f34358f1640 1 Processor -- start 2024-03-25T17:57:48.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.547+0000 7f34358f1640 1 -- start start 2024-03-25T17:57:48.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.547+0000 7f34358f1640 10 monclient: init 2024-03-25T17:57:48.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.547+0000 7f34358f1640 10 monclient: _reopen_session rank -1 2024-03-25T17:57:48.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.547+0000 7f34358f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:57:48.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.547+0000 7f34358f1640 10 monclient(hunting): picked mon.noname-c con 0x7f34300a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T17:57:48.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.547+0000 7f34358f1640 10 monclient(hunting): picked mon.noname-b con 0x7f343013f710 addr v1:172.21.15.18:6790/0 2024-03-25T17:57:48.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.547+0000 7f34358f1640 10 monclient(hunting): picked mon.noname-a con 0x7f34301430c0 addr v1:172.21.15.18:6789/0 2024-03-25T17:57:48.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.547+0000 7f34358f1640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3430142520 con 0x7f34301430c0 2024-03-25T17:57:48.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.547+0000 7f34358f1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34300b28a0 con 0x7f34300a98f0 2024-03-25T17:57:48.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.547+0000 7f34358f1640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34300b2a30 con 0x7f343013f710 2024-03-25T17:57:48.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.547+0000 7f34358f1640 10 monclient(hunting): _renew_subs 2024-03-25T17:57:48.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.547+0000 7f34358f1640 10 monclient(hunting): authenticate will time out at 6997.126624s 2024-03-25T17:57:48.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.554+0000 7f342e7fc640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f343013f710 0x7f34301429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:44664/0 (socket says 172.21.15.18:44664) 2024-03-25T17:57:48.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.554+0000 7f342e7fc640 1 -- 172.21.15.18:0/4263622957 learned_addr learned my addr 172.21.15.18:0/4263622957 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:57:48.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.574+0000 7f342dffb640 1 -- 172.21.15.18:0/4263622957 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231181186 0 0) 0x7f3430142520 con 0x7f34301430c0 2024-03-25T17:57:48.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.574+0000 7f342dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:57:48.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.574+0000 7f342dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:48.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.574+0000 7f342dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:48.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.574+0000 7f342dffb640 10 monclient(hunting): my global_id is 4284 2024-03-25T17:57:48.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.574+0000 7f342dffb640 1 -- 172.21.15.18:0/4263622957 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3418003520 con 0x7f34301430c0 2024-03-25T17:57:48.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.576+0000 7f342dffb640 1 -- 172.21.15.18:0/4263622957 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769174460 0 0) 0x7f34300b2a30 con 0x7f343013f710 2024-03-25T17:57:48.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.576+0000 7f342dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:57:48.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.576+0000 7f342dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:48.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.576+0000 7f342dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:48.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.576+0000 7f342dffb640 10 monclient(hunting): my global_id is 4292 2024-03-25T17:57:48.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.576+0000 7f342dffb640 1 -- 172.21.15.18:0/4263622957 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3430142520 con 0x7f343013f710 2024-03-25T17:57:48.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.581+0000 7f342dffb640 1 -- 172.21.15.18:0/4263622957 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453596433 0 0) 0x7f34300b28a0 con 0x7f34300a98f0 2024-03-25T17:57:48.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.581+0000 7f342dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:57:48.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.581+0000 7f342dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:48.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.581+0000 7f342dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:48.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.581+0000 7f342dffb640 10 monclient(hunting): my global_id is 4276 2024-03-25T17:57:48.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.581+0000 7f342dffb640 1 -- 172.21.15.18:0/4263622957 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34300b2a30 con 0x7f34300a98f0 2024-03-25T17:57:48.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.593+0000 7f342dffb640 1 -- 172.21.15.18:0/4263622957 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1045797472 0 0) 0x7f3418003520 con 0x7f34301430c0 2024-03-25T17:57:48.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.593+0000 7f342dffb640 1 -- 172.21.15.18:0/4263622957 >> v1:172.21.15.18:6790/0 conn(0x7f343013f710 legacy=0x7f34301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:48.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.593+0000 7f342dffb640 1 -- 172.21.15.18:0/4263622957 >> v1:172.21.15.169:6789/0 conn(0x7f34300a98f0 legacy=0x7f34300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:48.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.593+0000 7f342dffb640 10 monclient: _finish_hunting 0 2024-03-25T17:57:48.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.593+0000 7f342dffb640 1 monclient: found mon.noname-a 2024-03-25T17:57:48.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.593+0000 7f342dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:57:48.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.593+0000 7f342dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T17:57:48.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.593+0000 7f342dffb640 1 -- 172.21.15.18:0/4263622957 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34300b44c0 con 0x7f34301430c0 2024-03-25T17:57:48.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.593+0000 7f342dffb640 10 monclient: _finish_auth 0 2024-03-25T17:57:48.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.593+0000 7f342dffb640 10 monclient: _check_auth_tickets 2024-03-25T17:57:48.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.593+0000 7f342dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:57:48.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.593+0000 7f342dffb640 1 -- 172.21.15.18:0/4263622957 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3428001a10 con 0x7f34301430c0 2024-03-25T17:57:48.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.593+0000 7f34358f1640 5 monclient: authenticate success, global_id 4284 2024-03-25T17:57:48.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.593+0000 7f342dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:48.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.593+0000 7f342dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:57:48.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.593+0000 7f342dffb640 10 monclient: dump: 2024-03-25T17:57:48.595 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:48.595 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:48.595 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:48.595 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:48.595 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:48.595 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:48.595 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:48.595 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:48.596 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:48.596 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:48.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.593+0000 7f34358f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:57:48.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.615+0000 7f342dffb640 1 -- 172.21.15.18:0/4263622957 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3428001b70 con 0x7f34301430c0 2024-03-25T17:57:48.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.615+0000 7f342dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:57:48.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.615+0000 7f342dffb640 1 -- 172.21.15.18:0/4263622957 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3428001f80 con 0x7f34301430c0 2024-03-25T17:57:48.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.615+0000 7f34358f1640 10 monclient: get_monmap_and_config success 2024-03-25T17:57:48.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.615+0000 7f34358f1640 4 set_mon_vals no callback set 2024-03-25T17:57:48.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.615+0000 7f342dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:48.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.615+0000 7f342dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:57:48.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.615+0000 7f342dffb640 10 monclient: dump: 2024-03-25T17:57:48.617 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:48.617 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:48.617 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:48.618 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:48.618 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:48.618 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:48.618 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:48.618 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:48.618 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:48.618 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:48.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.616+0000 7f34358f1640 10 monclient: shutdown 2024-03-25T17:57:48.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.616+0000 7f34358f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:57:48.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.616+0000 7f34358f1640 1 -- 172.21.15.18:0/4263622957 >> v1:172.21.15.18:6789/0 conn(0x7f34301430c0 legacy=0x7f343014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:48.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.616+0000 7f34358f1640 1 -- 172.21.15.18:0/4263622957 shutdown_connections 2024-03-25T17:57:48.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.616+0000 7f34358f1640 1 -- 172.21.15.18:0/4263622957 >> 172.21.15.18:0/4263622957 conn(0x7f34300b7ea0 msgr2=0x7f34300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:57:48.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.616+0000 7f34358f1640 1 -- 172.21.15.18:0/4263622957 shutdown_connections 2024-03-25T17:57:48.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.616+0000 7f34358f1640 1 -- 172.21.15.18:0/4263622957 wait complete. 2024-03-25T17:57:48.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.616+0000 7f34358f1640 10 monclient: build_initial_monmap 2024-03-25T17:57:48.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.616+0000 7f34358f1640 10 monclient: monmap: 2024-03-25T17:57:48.620 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:57:48.620 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:57:48.620 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:57:48.620 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:57:48.620 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:57:48.620 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:57:48.620 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:57:48.621 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:57:48.621 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:57:48.621 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:48.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.617+0000 7f34358f1640 1 librados: starting msgr at 2024-03-25T17:57:48.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.617+0000 7f34358f1640 1 librados: starting objecter 2024-03-25T17:57:48.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.617+0000 7f34358f1640 1 Processor -- start 2024-03-25T17:57:48.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.617+0000 7f34358f1640 1 -- start start 2024-03-25T17:57:48.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.617+0000 7f34358f1640 1 librados: setting wanted keys 2024-03-25T17:57:48.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.617+0000 7f34358f1640 1 librados: calling monclient init 2024-03-25T17:57:48.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.617+0000 7f34358f1640 10 monclient: init 2024-03-25T17:57:48.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.618+0000 7f34358f1640 10 monclient: _reopen_session rank -1 2024-03-25T17:57:48.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.618+0000 7f34358f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:57:48.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.618+0000 7f34358f1640 10 monclient(hunting): picked mon.noname-b con 0x7f34300a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T17:57:48.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.618+0000 7f34358f1640 10 monclient(hunting): picked mon.noname-c con 0x7f343013f710 addr v1:172.21.15.18:6790/0 2024-03-25T17:57:48.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.618+0000 7f34358f1640 10 monclient(hunting): picked mon.noname-a con 0x7f34301430c0 addr v1:172.21.15.18:6789/0 2024-03-25T17:57:48.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.618+0000 7f34358f1640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34300abb00 con 0x7f34301430c0 2024-03-25T17:57:48.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.618+0000 7f34358f1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34300abde0 con 0x7f34300a98f0 2024-03-25T17:57:48.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.618+0000 7f34358f1640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3430138e20 con 0x7f343013f710 2024-03-25T17:57:48.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.618+0000 7f34358f1640 10 monclient(hunting): _renew_subs 2024-03-25T17:57:48.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.618+0000 7f34358f1640 10 monclient(hunting): authenticate will time out at 6997.196932s 2024-03-25T17:57:48.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.627+0000 7f342effd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f34300a98f0 0x7f34301385e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:55756/0 (socket says 172.21.15.18:55756) 2024-03-25T17:57:48.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.627+0000 7f342effd640 1 -- 172.21.15.18:0/160763707 learned_addr learned my addr 172.21.15.18:0/160763707 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:57:48.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.629+0000 7f3417fff640 10 client.?.objecter ms_handle_connect 0x7f34300a98f0 2024-03-25T17:57:48.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.629+0000 7f3417fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:48.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.632+0000 7f3417fff640 10 client.?.objecter ms_handle_connect 0x7f34301430c0 2024-03-25T17:57:48.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.632+0000 7f3417fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:48.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.642+0000 7f3417fff640 10 client.?.objecter ms_handle_connect 0x7f343013f710 2024-03-25T17:57:48.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.642+0000 7f3417fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:48.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.644+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086397061 0 0) 0x7f34300abde0 con 0x7f34300a98f0 2024-03-25T17:57:48.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.644+0000 7f3417fff640 10 monclient(hunting): _negotiate 2024-03-25T17:57:48.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.644+0000 7f3417fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:48.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.644+0000 7f3417fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:48.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.644+0000 7f3417fff640 10 monclient(hunting): my global_id is 4279 2024-03-25T17:57:48.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.644+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f341c004eb0 con 0x7f34300a98f0 2024-03-25T17:57:48.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.652+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478355683 0 0) 0x7f34300abb00 con 0x7f34301430c0 2024-03-25T17:57:48.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.652+0000 7f3417fff640 10 monclient(hunting): _negotiate 2024-03-25T17:57:48.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.652+0000 7f3417fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:48.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.652+0000 7f3417fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.652+0000 7f3417fff640 10 monclient(hunting): my global_id is 4287 2024-03-25T17:57:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.652+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34300abde0 con 0x7f34301430c0 2024-03-25T17:57:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.653+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605450872 0 0) 0x7f3430138e20 con 0x7f343013f710 2024-03-25T17:57:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.653+0000 7f3417fff640 10 monclient(hunting): _negotiate 2024-03-25T17:57:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.653+0000 7f3417fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.653+0000 7f3417fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.653+0000 7f3417fff640 10 monclient(hunting): my global_id is 4295 2024-03-25T17:57:48.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.653+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34300abb00 con 0x7f343013f710 2024-03-25T17:57:48.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.664+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 589549356 0 0) 0x7f341c004eb0 con 0x7f34300a98f0 2024-03-25T17:57:48.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.664+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3430138e20 con 0x7f34300a98f0 2024-03-25T17:57:48.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.664+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3420001a10 con 0x7f34300a98f0 2024-03-25T17:57:48.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.664+0000 7f3417fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:48.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.664+0000 7f3417fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:57:48.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.664+0000 7f3417fff640 10 monclient(hunting): dump: 2024-03-25T17:57:48.665 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:48.665 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:48.665 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:48.665 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:48.665 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:48.665 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:48.666 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:48.666 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:48.666 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:48.666 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:48.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.664+0000 7f3417fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:57:48.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.671+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1677465331 0 0) 0x7f34300abb00 con 0x7f343013f710 2024-03-25T17:57:48.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.671+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f341c004eb0 con 0x7f343013f710 2024-03-25T17:57:48.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.671+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3424001930 con 0x7f343013f710 2024-03-25T17:57:48.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.671+0000 7f3417fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:48.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.671+0000 7f3417fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:57:48.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.671+0000 7f3417fff640 10 monclient(hunting): dump: 2024-03-25T17:57:48.672 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:48.672 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:48.672 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:48.672 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:48.672 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:48.672 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:48.672 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:48.672 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:48.672 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:48.672 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:48.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.674+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2222680966 0 0) 0x7f34300abde0 con 0x7f34301430c0 2024-03-25T17:57:48.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.674+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34300abb00 con 0x7f34301430c0 2024-03-25T17:57:48.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.674+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3428001e80 con 0x7f34301430c0 2024-03-25T17:57:48.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.674+0000 7f3417fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:48.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.674+0000 7f3417fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:57:48.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.674+0000 7f3417fff640 10 monclient(hunting): dump: 2024-03-25T17:57:48.675 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:48.675 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:48.675 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:48.675 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:48.675 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:48.676 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:48.676 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:48.676 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:48.676 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:48.676 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:48.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4219756300 0 0) 0x7f3430138e20 con 0x7f34300a98f0 2024-03-25T17:57:48.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 >> v1:172.21.15.18:6790/0 conn(0x7f343013f710 legacy=0x7f3430136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:48.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 >> v1:172.21.15.18:6789/0 conn(0x7f34301430c0 legacy=0x7f34300ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:48.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f3417fff640 10 monclient: _finish_hunting 0 2024-03-25T17:57:48.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f3417fff640 1 monclient: found mon.b 2024-03-25T17:57:48.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f3417fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:57:48.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f3417fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:57:48.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3430137b00 con 0x7f34300a98f0 2024-03-25T17:57:48.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f3417fff640 10 monclient: _finish_auth 0 2024-03-25T17:57:48.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f3417fff640 10 monclient: _check_auth_tickets 2024-03-25T17:57:48.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f3417fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:57:48.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f34358f1640 5 monclient: authenticate success, global_id 4279 2024-03-25T17:57:48.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f34358f1640 10 monclient: _renew_subs 2024-03-25T17:57:48.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f34358f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:57:48.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f34358f1640 1 -- 172.21.15.18:0/160763707 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34301e34b0 con 0x7f34300a98f0 2024-03-25T17:57:48.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f34358f1640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:57:48.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f34358f1640 10 monclient: _renew_subs 2024-03-25T17:57:48.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f34358f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:57:48.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f34358f1640 1 -- 172.21.15.18:0/160763707 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34301e3820 con 0x7f34300a98f0 2024-03-25T17:57:48.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.681+0000 7f34358f1640 1 librados: init done 2024-03-25T17:57:48.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.682+0000 7f34358f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:48.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.682+0000 7f34358f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:48.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.682+0000 7f34358f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:57:48.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.682+0000 7f34358f1640 1 -- 172.21.15.18:0/160763707 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3430138fb0 con 0x7f34300a98f0 2024-03-25T17:57:48.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.973+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3420001c70 con 0x7f34300a98f0 2024-03-25T17:57:48.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.974+0000 7f3417fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:57:48.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.974+0000 7f342d7fa640 4 set_mon_vals no callback set 2024-03-25T17:57:48.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.974+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3420003a60 con 0x7f34300a98f0 2024-03-25T17:57:48.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.974+0000 7f3417fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:48.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.974+0000 7f3417fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:57:48.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.974+0000 7f3417fff640 10 monclient: dump: 2024-03-25T17:57:48.975 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:48.975 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:48.975 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:48.975 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:48.975 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:48.976 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:48.976 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:48.976 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:48.976 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:48.976 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:48.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.975+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f3420003c80 con 0x7f34300a98f0 2024-03-25T17:57:48.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.975+0000 7f342e7fc640 10 monclient: get_auth_request con 0x7f341c06e0b0 auth_method 0 2024-03-25T17:57:48.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.975+0000 7f3417fff640 10 client.4279.objecter ms_handle_connect 0x7f341c06e0b0 2024-03-25T17:57:48.982 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.981+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f34200548b0 con 0x7f34300a98f0 2024-03-25T17:57:48.982 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.981+0000 7f3417fff640 10 client.4279.objecter ms_dispatch 0x7f3430000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:57:48.982 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.981+0000 7f3417fff640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:57:48.982 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.981+0000 7f3417fff640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:57:48.982 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.981+0000 7f3417fff640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-25T17:57:48.982 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.982+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f34200503c0 con 0x7f34300a98f0 2024-03-25T17:57:48.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.982+0000 7f3417fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:48.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:48.982+0000 7f3417fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:57:49.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.202+0000 7f34358f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:49.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.202+0000 7f34358f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:49.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.202+0000 7f34358f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:57:49.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.202+0000 7f34358f1640 1 -- 172.21.15.18:0/160763707 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34300627b0 con 0x7f34300a98f0 2024-03-25T17:57:49.405 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.404+0000 7f3417fff640 1 -- 172.21.15.18:0/160763707 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f34200522f0 con 0x7f34300a98f0 2024-03-25T17:57:49.405 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.404+0000 7f3417fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:49.405 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.404+0000 7f3417fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:57:49.405 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:57:49.405 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:57:49.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.406+0000 7f34358f1640 10 librados: watch_flush enter 2024-03-25T17:57:49.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.406+0000 7f34358f1640 10 librados: watch_flush exit 2024-03-25T17:57:49.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.406+0000 7f34358f1640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-25T17:57:49.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.406+0000 7f34358f1640 10 client.4279.objecter successfully canceled tick 2024-03-25T17:57:49.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.407+0000 7f34358f1640 1 -- 172.21.15.18:0/160763707 >> v1:172.21.15.18:6800/3721244451 conn(0x7f341c06e0b0 legacy=0x7f341c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:49.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.407+0000 7f34358f1640 10 monclient: shutdown 2024-03-25T17:57:49.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.407+0000 7f34358f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:57:49.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.407+0000 7f34358f1640 1 -- 172.21.15.18:0/160763707 >> v1:172.21.15.169:6789/0 conn(0x7f34300a98f0 legacy=0x7f34301385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:49.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.407+0000 7f34358f1640 1 -- 172.21.15.18:0/160763707 shutdown_connections 2024-03-25T17:57:49.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.407+0000 7f34358f1640 1 -- 172.21.15.18:0/160763707 >> 172.21.15.18:0/160763707 conn(0x7f34300b7ea0 msgr2=0x7f343014ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:57:49.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.407+0000 7f34358f1640 1 -- 172.21.15.18:0/160763707 shutdown_connections 2024-03-25T17:57:49.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.407+0000 7f34358f1640 1 -- 172.21.15.18:0/160763707 wait complete. 2024-03-25T17:57:49.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:49.407+0000 7f34358f1640 1 librados: shutdown 2024-03-25T17:57:49.419 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:57:56.227 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:57:56.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.343+0000 7fb2c9064640 10 monclient: get_monmap_and_config 2024-03-25T17:57:56.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.343+0000 7fb2c9064640 10 monclient: build_initial_monmap 2024-03-25T17:57:56.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.343+0000 7fb2c9064640 10 monclient: monmap: 2024-03-25T17:57:56.344 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:57:56.344 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:56.344 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:57:56.344559+0000 2024-03-25T17:57:56.344 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:57:56.344559+0000 2024-03-25T17:57:56.344 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:57:56.345 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:56.345 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:57:56.345 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:57:56.345 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:57:56.345 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:56.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.343+0000 7fb2c9064640 1 Processor -- start 2024-03-25T17:57:56.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.343+0000 7fb2c9064640 1 -- start start 2024-03-25T17:57:56.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.343+0000 7fb2c9064640 10 monclient: init 2024-03-25T17:57:56.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.344+0000 7fb2c9064640 10 monclient: _reopen_session rank -1 2024-03-25T17:57:56.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.344+0000 7fb2c9064640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:57:56.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.344+0000 7fb2c9064640 10 monclient(hunting): picked mon.noname-a con 0x7fb2c40a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T17:57:56.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.344+0000 7fb2c9064640 10 monclient(hunting): picked mon.noname-b con 0x7fb2c413f710 addr v1:172.21.15.18:6790/0 2024-03-25T17:57:56.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.344+0000 7fb2c9064640 10 monclient(hunting): picked mon.noname-c con 0x7fb2c41430c0 addr v1:172.21.15.169:6789/0 2024-03-25T17:57:56.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.344+0000 7fb2c9064640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c4142520 con 0x7fb2c40a98f0 2024-03-25T17:57:56.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.344+0000 7fb2c9064640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c40b28a0 con 0x7fb2c41430c0 2024-03-25T17:57:56.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.344+0000 7fb2c9064640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c40b2a30 con 0x7fb2c413f710 2024-03-25T17:57:56.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.344+0000 7fb2c9064640 10 monclient(hunting): _renew_subs 2024-03-25T17:57:56.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.344+0000 7fb2c9064640 10 monclient(hunting): authenticate will time out at 7004.923263s 2024-03-25T17:57:56.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.348+0000 7fb2c359e640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fb2c41430c0 0x7fb2c414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:46298/0 (socket says 172.21.15.18:46298) 2024-03-25T17:57:56.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.348+0000 7fb2c359e640 1 -- 172.21.15.18:0/1297102764 learned_addr learned my addr 172.21.15.18:0/1297102764 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:57:56.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.363+0000 7fb2c1d9b640 1 -- 172.21.15.18:0/1297102764 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543502480 0 0) 0x7fb2c40b28a0 con 0x7fb2c41430c0 2024-03-25T17:57:56.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.363+0000 7fb2c1d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:57:56.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.363+0000 7fb2c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:56.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.363+0000 7fb2c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:56.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.363+0000 7fb2c1d9b640 10 monclient(hunting): my global_id is 4282 2024-03-25T17:57:56.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.363+0000 7fb2c1d9b640 1 -- 172.21.15.18:0/1297102764 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2ac003520 con 0x7fb2c41430c0 2024-03-25T17:57:56.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.370+0000 7fb2c1d9b640 1 -- 172.21.15.18:0/1297102764 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506965103 0 0) 0x7fb2c4142520 con 0x7fb2c40a98f0 2024-03-25T17:57:56.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.370+0000 7fb2c1d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:57:56.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.370+0000 7fb2c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:56.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.370+0000 7fb2c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:56.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.370+0000 7fb2c1d9b640 10 monclient(hunting): my global_id is 4290 2024-03-25T17:57:56.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.370+0000 7fb2c1d9b640 1 -- 172.21.15.18:0/1297102764 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c40b28a0 con 0x7fb2c40a98f0 2024-03-25T17:57:56.375 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c1d9b640 1 -- 172.21.15.18:0/1297102764 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2865441242 0 0) 0x7fb2ac003520 con 0x7fb2c41430c0 2024-03-25T17:57:56.375 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c1d9b640 1 -- 172.21.15.18:0/1297102764 >> v1:172.21.15.18:6790/0 conn(0x7fb2c413f710 legacy=0x7fb2c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:56.375 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c1d9b640 1 -- 172.21.15.18:0/1297102764 >> v1:172.21.15.18:6789/0 conn(0x7fb2c40a98f0 legacy=0x7fb2c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:56.375 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c1d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:57:56.375 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c1d9b640 1 monclient: found mon.noname-c 2024-03-25T17:57:56.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:57:56.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T17:57:56.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c1d9b640 1 -- 172.21.15.18:0/1297102764 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c40b44c0 con 0x7fb2c41430c0 2024-03-25T17:57:56.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c1d9b640 10 monclient: _finish_auth 0 2024-03-25T17:57:56.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c1d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:57:56.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:57:56.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c1d9b640 1 -- 172.21.15.18:0/1297102764 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fb2bc001a10 con 0x7fb2c41430c0 2024-03-25T17:57:56.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c9064640 5 monclient: authenticate success, global_id 4282 2024-03-25T17:57:56.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c9064640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:57:56.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:56.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:57:56.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c1d9b640 10 monclient: dump: 2024-03-25T17:57:56.376 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:56.376 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:56.377 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:56.377 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:56.377 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:56.377 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:56.377 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:56.377 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:56.377 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:56.377 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:56.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.374+0000 7fb2c9064640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:57:56.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.387+0000 7fb2c1d9b640 1 -- 172.21.15.18:0/1297102764 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2bc001b70 con 0x7fb2c41430c0 2024-03-25T17:57:56.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.387+0000 7fb2c1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:57:56.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.387+0000 7fb2c1d9b640 1 -- 172.21.15.18:0/1297102764 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fb2bc001f80 con 0x7fb2c41430c0 2024-03-25T17:57:56.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.387+0000 7fb2c9064640 10 monclient: get_monmap_and_config success 2024-03-25T17:57:56.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.387+0000 7fb2c9064640 4 set_mon_vals no callback set 2024-03-25T17:57:56.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.387+0000 7fb2c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:56.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.387+0000 7fb2c1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:57:56.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.387+0000 7fb2c1d9b640 10 monclient: dump: 2024-03-25T17:57:56.388 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:56.388 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:56.388 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:56.388 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:56.388 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:56.388 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:56.388 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:56.388 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:56.388 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:56.389 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:56.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.387+0000 7fb2c9064640 10 monclient: shutdown 2024-03-25T17:57:56.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.387+0000 7fb2c9064640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:57:56.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.387+0000 7fb2c9064640 1 -- 172.21.15.18:0/1297102764 >> v1:172.21.15.169:6789/0 conn(0x7fb2c41430c0 legacy=0x7fb2c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:56.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.387+0000 7fb2c9064640 1 -- 172.21.15.18:0/1297102764 shutdown_connections 2024-03-25T17:57:56.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.388+0000 7fb2c9064640 1 -- 172.21.15.18:0/1297102764 >> 172.21.15.18:0/1297102764 conn(0x7fb2c40b7ea0 msgr2=0x7fb2c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:57:56.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.388+0000 7fb2c9064640 1 -- 172.21.15.18:0/1297102764 shutdown_connections 2024-03-25T17:57:56.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.388+0000 7fb2c9064640 1 -- 172.21.15.18:0/1297102764 wait complete. 2024-03-25T17:57:56.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.388+0000 7fb2c9064640 10 monclient: build_initial_monmap 2024-03-25T17:57:56.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.388+0000 7fb2c9064640 10 monclient: monmap: 2024-03-25T17:57:56.391 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:57:56.391 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:57:56.391 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:57:56.391 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:57:56.391 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:57:56.392 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:57:56.392 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:57:56.392 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:57:56.392 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:57:56.392 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:56.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.389+0000 7fb2c9064640 1 librados: starting msgr at 2024-03-25T17:57:56.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.389+0000 7fb2c9064640 1 librados: starting objecter 2024-03-25T17:57:56.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.389+0000 7fb2c9064640 1 Processor -- start 2024-03-25T17:57:56.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.389+0000 7fb2c9064640 1 -- start start 2024-03-25T17:57:56.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.389+0000 7fb2c9064640 1 librados: setting wanted keys 2024-03-25T17:57:56.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.389+0000 7fb2c9064640 1 librados: calling monclient init 2024-03-25T17:57:56.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.389+0000 7fb2c9064640 10 monclient: init 2024-03-25T17:57:56.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.390+0000 7fb2c9064640 10 monclient: _reopen_session rank -1 2024-03-25T17:57:56.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.390+0000 7fb2c9064640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:57:56.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.390+0000 7fb2c9064640 10 monclient(hunting): picked mon.noname-c con 0x7fb2c40a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T17:57:56.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.390+0000 7fb2c9064640 10 monclient(hunting): picked mon.noname-b con 0x7fb2c413f710 addr v1:172.21.15.169:6789/0 2024-03-25T17:57:56.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.390+0000 7fb2c9064640 10 monclient(hunting): picked mon.noname-a con 0x7fb2c41430c0 addr v1:172.21.15.18:6789/0 2024-03-25T17:57:56.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.390+0000 7fb2c9064640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c413c2b0 con 0x7fb2c41430c0 2024-03-25T17:57:56.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.390+0000 7fb2c9064640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c4136f10 con 0x7fb2c413f710 2024-03-25T17:57:56.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.390+0000 7fb2c9064640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c41370a0 con 0x7fb2c40a98f0 2024-03-25T17:57:56.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.390+0000 7fb2c9064640 10 monclient(hunting): _renew_subs 2024-03-25T17:57:56.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.390+0000 7fb2c9064640 10 monclient(hunting): authenticate will time out at 7004.969023s 2024-03-25T17:57:56.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.401+0000 7fb2c259c640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fb2c413f710 0x7fb2c41377d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:46342/0 (socket says 172.21.15.18:46342) 2024-03-25T17:57:56.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.401+0000 7fb2c259c640 1 -- 172.21.15.18:0/861307391 learned_addr learned my addr 172.21.15.18:0/861307391 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:57:56.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.407+0000 7fb2ab7fe640 10 client.?.objecter ms_handle_connect 0x7fb2c413f710 2024-03-25T17:57:56.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.407+0000 7fb2ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:56.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.420+0000 7fb2ab7fe640 10 client.?.objecter ms_handle_connect 0x7fb2c41430c0 2024-03-25T17:57:56.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.420+0000 7fb2ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:56.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.427+0000 7fb2ab7fe640 10 client.?.objecter ms_handle_connect 0x7fb2c40a98f0 2024-03-25T17:57:56.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.427+0000 7fb2ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:57:56.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.429+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2360794120 0 0) 0x7fb2c413c2b0 con 0x7fb2c41430c0 2024-03-25T17:57:56.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.429+0000 7fb2ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:57:56.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.429+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:56.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.429+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:56.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.429+0000 7fb2ab7fe640 10 monclient(hunting): my global_id is 4293 2024-03-25T17:57:56.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.429+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b0004eb0 con 0x7fb2c41430c0 2024-03-25T17:57:56.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.430+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045260652 0 0) 0x7fb2c4136f10 con 0x7fb2c413f710 2024-03-25T17:57:56.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.430+0000 7fb2ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:57:56.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.430+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:56.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.430+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:56.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.430+0000 7fb2ab7fe640 10 monclient(hunting): my global_id is 4285 2024-03-25T17:57:56.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.430+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c413c2b0 con 0x7fb2c413f710 2024-03-25T17:57:56.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.444+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672039152 0 0) 0x7fb2c41370a0 con 0x7fb2c40a98f0 2024-03-25T17:57:56.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.444+0000 7fb2ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:57:56.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.445+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:57:56.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.445+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:57:56.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.445+0000 7fb2ab7fe640 10 monclient(hunting): my global_id is 4301 2024-03-25T17:57:56.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.445+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c4136f10 con 0x7fb2c40a98f0 2024-03-25T17:57:56.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.457+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4080478724 0 0) 0x7fb2b0004eb0 con 0x7fb2c41430c0 2024-03-25T17:57:56.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.457+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2c41370a0 con 0x7fb2c41430c0 2024-03-25T17:57:56.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.457+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fb2bc001e80 con 0x7fb2c41430c0 2024-03-25T17:57:56.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.457+0000 7fb2ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:56.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.457+0000 7fb2ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:57:56.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.457+0000 7fb2ab7fe640 10 monclient(hunting): dump: 2024-03-25T17:57:56.458 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:56.458 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:56.459 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:56.459 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:56.459 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:56.459 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:56.459 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:56.459 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:56.459 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:56.459 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:56.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.457+0000 7fb2ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:57:56.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.466+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 693945606 0 0) 0x7fb2c4136f10 con 0x7fb2c40a98f0 2024-03-25T17:57:56.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.466+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2b0004eb0 con 0x7fb2c40a98f0 2024-03-25T17:57:56.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.466+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fb2b4001930 con 0x7fb2c40a98f0 2024-03-25T17:57:56.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.466+0000 7fb2ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:56.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.466+0000 7fb2ab7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:57:56.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.466+0000 7fb2ab7fe640 10 monclient(hunting): dump: 2024-03-25T17:57:56.468 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:56.468 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:56.468 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:56.468 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:56.468 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:56.468 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:56.468 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:56.468 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:56.468 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:56.468 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:56.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.467+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3965342808 0 0) 0x7fb2c413c2b0 con 0x7fb2c413f710 2024-03-25T17:57:56.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.467+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2c4136f10 con 0x7fb2c413f710 2024-03-25T17:57:56.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.467+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fb2b80019b0 con 0x7fb2c413f710 2024-03-25T17:57:56.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.467+0000 7fb2ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:56.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.467+0000 7fb2ab7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:57:56.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.467+0000 7fb2ab7fe640 10 monclient(hunting): dump: 2024-03-25T17:57:56.469 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:56.470 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:56.470 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:56.470 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:56.470 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:56.470 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:56.470 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:56.470 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:56.470 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:56.470 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:56.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2055576725 0 0) 0x7fb2c41370a0 con 0x7fb2c41430c0 2024-03-25T17:57:56.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 >> v1:172.21.15.18:6790/0 conn(0x7fb2c40a98f0 legacy=0x7fb2c41d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:56.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 >> v1:172.21.15.169:6789/0 conn(0x7fb2c413f710 legacy=0x7fb2c41377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:56.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:57:56.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2ab7fe640 1 monclient: found mon.a 2024-03-25T17:57:56.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:57:56.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2ab7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:57:56.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c4137230 con 0x7fb2c41430c0 2024-03-25T17:57:56.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2ab7fe640 10 monclient: _finish_auth 0 2024-03-25T17:57:56.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:57:56.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:57:56.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2c9064640 5 monclient: authenticate success, global_id 4293 2024-03-25T17:57:56.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2c9064640 10 monclient: _renew_subs 2024-03-25T17:57:56.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2c9064640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:57:56.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2c9064640 1 -- 172.21.15.18:0/861307391 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c413b620 con 0x7fb2c41430c0 2024-03-25T17:57:56.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2c9064640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:57:56.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2c9064640 10 monclient: _renew_subs 2024-03-25T17:57:56.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2c9064640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:57:56.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2c9064640 1 -- 172.21.15.18:0/861307391 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c41338f0 con 0x7fb2c41430c0 2024-03-25T17:57:56.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.470+0000 7fb2c9064640 1 librados: init done 2024-03-25T17:57:56.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.471+0000 7fb2c9064640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:56.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.471+0000 7fb2c9064640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:56.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.471+0000 7fb2c9064640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:57:56.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.471+0000 7fb2c9064640 1 -- 172.21.15.18:0/861307391 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c413b9c0 con 0x7fb2c41430c0 2024-03-25T17:57:56.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.837+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2bc0020e0 con 0x7fb2c41430c0 2024-03-25T17:57:56.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.837+0000 7fb2ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:57:56.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.837+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fb2bc004e10 con 0x7fb2c41430c0 2024-03-25T17:57:56.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.837+0000 7fb2ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:57:56.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.837+0000 7fb2ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:57:56.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.837+0000 7fb2ab7fe640 10 monclient: dump: 2024-03-25T17:57:56.838 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:57:56.838 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:57:56.838 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:56.838 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:57:56.838 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:57:56.838 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:57:56.838 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:57:56.839 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:57:56.839 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:57:56.839 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:57:56.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.837+0000 7fb2c159a640 4 set_mon_vals no callback set 2024-03-25T17:57:56.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.838+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fb2bc01ade0 con 0x7fb2c41430c0 2024-03-25T17:57:56.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.838+0000 7fb2c2d9d640 10 monclient: get_auth_request con 0x7fb2b006e0f0 auth_method 0 2024-03-25T17:57:56.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.839+0000 7fb2ab7fe640 10 client.4293.objecter ms_handle_connect 0x7fb2b006e0f0 2024-03-25T17:57:56.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.844+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fb2bc054910 con 0x7fb2c41430c0 2024-03-25T17:57:56.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.844+0000 7fb2ab7fe640 10 client.4293.objecter ms_dispatch 0x7fb2c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:57:56.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.844+0000 7fb2ab7fe640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:57:56.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.844+0000 7fb2ab7fe640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:57:56.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.844+0000 7fb2ab7fe640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-25T17:57:56.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.844+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2bc050420 con 0x7fb2c41430c0 2024-03-25T17:57:56.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.845+0000 7fb2ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:57:56.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:56.845+0000 7fb2ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:57:57.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.078+0000 7fb2c9064640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:57.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.078+0000 7fb2c9064640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:57.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.078+0000 7fb2c9064640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:57:57.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.078+0000 7fb2c9064640 1 -- 172.21.15.18:0/861307391 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2c40627b0 con 0x7fb2c41430c0 2024-03-25T17:57:57.144 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.143+0000 7fb2ab7fe640 1 -- 172.21.15.18:0/861307391 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fb2bc052350 con 0x7fb2c41430c0 2024-03-25T17:57:57.144 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.143+0000 7fb2ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:57:57.144 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.143+0000 7fb2ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:57:57.145 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:57:57.145 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:57:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.146+0000 7fb2c9064640 10 librados: watch_flush enter 2024-03-25T17:57:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.146+0000 7fb2c9064640 10 librados: watch_flush exit 2024-03-25T17:57:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.146+0000 7fb2c9064640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-25T17:57:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.146+0000 7fb2c9064640 10 client.4293.objecter successfully canceled tick 2024-03-25T17:57:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.146+0000 7fb2c9064640 1 -- 172.21.15.18:0/861307391 >> v1:172.21.15.18:6800/3721244451 conn(0x7fb2b006e0f0 legacy=0x7fb2b00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.146+0000 7fb2c9064640 10 monclient: shutdown 2024-03-25T17:57:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.146+0000 7fb2c9064640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:57:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.146+0000 7fb2c9064640 1 -- 172.21.15.18:0/861307391 >> v1:172.21.15.18:6789/0 conn(0x7fb2c41430c0 legacy=0x7fb2c4133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:57:57.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.146+0000 7fb2c9064640 1 -- 172.21.15.18:0/861307391 shutdown_connections 2024-03-25T17:57:57.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.146+0000 7fb2c9064640 1 -- 172.21.15.18:0/861307391 >> 172.21.15.18:0/861307391 conn(0x7fb2c40b7ea0 msgr2=0x7fb2c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:57:57.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.146+0000 7fb2c9064640 1 -- 172.21.15.18:0/861307391 shutdown_connections 2024-03-25T17:57:57.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.146+0000 7fb2c9064640 1 -- 172.21.15.18:0/861307391 wait complete. 2024-03-25T17:57:57.148 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:57:57.146+0000 7fb2c9064640 1 librados: shutdown 2024-03-25T17:57:57.159 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:58:01.903 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:59.696 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:01.907 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:59.700 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:01.937 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:01:59.730 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:01.958 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:59.707 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:01.962 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:59.756 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:01.966 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:59.759 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:01.968 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:59.717 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:01.999 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:01:59.749 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:02.001 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:01:59.795 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:03.966 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:58:04.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.081+0000 7fe7e1977640 10 monclient: get_monmap_and_config 2024-03-25T17:58:04.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.081+0000 7fe7e1977640 10 monclient: build_initial_monmap 2024-03-25T17:58:04.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.081+0000 7fe7e1977640 10 monclient: monmap: 2024-03-25T17:58:04.082 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:58:04.082 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:04.082 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:58:04.082842+0000 2024-03-25T17:58:04.082 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:58:04.082842+0000 2024-03-25T17:58:04.082 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:58:04.082 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:04.083 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:58:04.083 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:58:04.083 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:58:04.083 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:04.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.082+0000 7fe7e1977640 1 Processor -- start 2024-03-25T17:58:04.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.082+0000 7fe7e1977640 1 -- start start 2024-03-25T17:58:04.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.082+0000 7fe7e1977640 10 monclient: init 2024-03-25T17:58:04.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.082+0000 7fe7e1977640 10 monclient: _reopen_session rank -1 2024-03-25T17:58:04.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.082+0000 7fe7e1977640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:58:04.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.082+0000 7fe7e1977640 10 monclient(hunting): picked mon.noname-a con 0x7fe7dc0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T17:58:04.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.082+0000 7fe7e1977640 10 monclient(hunting): picked mon.noname-c con 0x7fe7dc13f710 addr v1:172.21.15.169:6789/0 2024-03-25T17:58:04.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.082+0000 7fe7e1977640 10 monclient(hunting): picked mon.noname-b con 0x7fe7dc1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T17:58:04.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.083+0000 7fe7e1977640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7dc142520 con 0x7fe7dc0a98f0 2024-03-25T17:58:04.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.083+0000 7fe7e1977640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7dc0b28a0 con 0x7fe7dc13f710 2024-03-25T17:58:04.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.083+0000 7fe7e1977640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7dc0b2a30 con 0x7fe7dc1430c0 2024-03-25T17:58:04.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.083+0000 7fe7e1977640 10 monclient(hunting): _renew_subs 2024-03-25T17:58:04.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.083+0000 7fe7e1977640 10 monclient(hunting): authenticate will time out at 7012.661707s 2024-03-25T17:58:04.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.093+0000 7fe7daffd640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fe7dc0a98f0 0x7fe7dc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:36526/0 (socket says 172.21.15.18:36526) 2024-03-25T17:58:04.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.093+0000 7fe7daffd640 1 -- 172.21.15.18:0/3048362600 learned_addr learned my addr 172.21.15.18:0/3048362600 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:58:04.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.104+0000 7fe7d9ffb640 1 -- 172.21.15.18:0/3048362600 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958570190 0 0) 0x7fe7dc142520 con 0x7fe7dc0a98f0 2024-03-25T17:58:04.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.104+0000 7fe7d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:58:04.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.104+0000 7fe7d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:04.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.104+0000 7fe7d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:04.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.104+0000 7fe7d9ffb640 10 monclient(hunting): my global_id is 4296 2024-03-25T17:58:04.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.104+0000 7fe7d9ffb640 1 -- 172.21.15.18:0/3048362600 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c4003520 con 0x7fe7dc0a98f0 2024-03-25T17:58:04.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.106+0000 7fe7d9ffb640 1 -- 172.21.15.18:0/3048362600 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999161386 0 0) 0x7fe7dc0b28a0 con 0x7fe7dc13f710 2024-03-25T17:58:04.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.106+0000 7fe7d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:58:04.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.106+0000 7fe7d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:04.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.106+0000 7fe7d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:04.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.106+0000 7fe7d9ffb640 10 monclient(hunting): my global_id is 4288 2024-03-25T17:58:04.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.106+0000 7fe7d9ffb640 1 -- 172.21.15.18:0/3048362600 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7dc142520 con 0x7fe7dc13f710 2024-03-25T17:58:04.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.112+0000 7fe7d9ffb640 1 -- 172.21.15.18:0/3048362600 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2653873814 0 0) 0x7fe7c4003520 con 0x7fe7dc0a98f0 2024-03-25T17:58:04.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.112+0000 7fe7d9ffb640 1 -- 172.21.15.18:0/3048362600 >> v1:172.21.15.18:6790/0 conn(0x7fe7dc1430c0 legacy=0x7fe7dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:04.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.112+0000 7fe7d9ffb640 1 -- 172.21.15.18:0/3048362600 >> v1:172.21.15.169:6789/0 conn(0x7fe7dc13f710 legacy=0x7fe7dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:04.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.112+0000 7fe7d9ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:58:04.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.112+0000 7fe7d9ffb640 1 monclient: found mon.noname-a 2024-03-25T17:58:04.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.112+0000 7fe7d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:58:04.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.112+0000 7fe7d9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T17:58:04.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.112+0000 7fe7d9ffb640 1 -- 172.21.15.18:0/3048362600 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7dc0b44c0 con 0x7fe7dc0a98f0 2024-03-25T17:58:04.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.112+0000 7fe7d9ffb640 10 monclient: _finish_auth 0 2024-03-25T17:58:04.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.112+0000 7fe7d9ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:58:04.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.112+0000 7fe7d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:58:04.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.112+0000 7fe7d9ffb640 1 -- 172.21.15.18:0/3048362600 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fe7cc001690 con 0x7fe7dc0a98f0 2024-03-25T17:58:04.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.112+0000 7fe7d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:04.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.112+0000 7fe7d9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:58:04.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.112+0000 7fe7d9ffb640 10 monclient: dump: 2024-03-25T17:58:04.114 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:04.114 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:04.114 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:04.114 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:04.114 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:04.114 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:04.114 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:04.114 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:04.114 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:04.115 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:04.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.112+0000 7fe7e1977640 5 monclient: authenticate success, global_id 4296 2024-03-25T17:58:04.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.112+0000 7fe7e1977640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:58:04.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.117+0000 7fe7d9ffb640 1 -- 172.21.15.18:0/3048362600 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7cc0017f0 con 0x7fe7dc0a98f0 2024-03-25T17:58:04.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.117+0000 7fe7d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:58:04.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.117+0000 7fe7d9ffb640 1 -- 172.21.15.18:0/3048362600 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fe7cc001c00 con 0x7fe7dc0a98f0 2024-03-25T17:58:04.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.117+0000 7fe7d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:04.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.117+0000 7fe7d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:58:04.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.117+0000 7fe7d9ffb640 10 monclient: dump: 2024-03-25T17:58:04.118 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:04.118 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:04.119 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:04.119 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:04.119 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:04.119 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:04.119 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:04.119 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:04.119 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:04.119 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:04.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.117+0000 7fe7e1977640 10 monclient: get_monmap_and_config success 2024-03-25T17:58:04.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.117+0000 7fe7e1977640 4 set_mon_vals no callback set 2024-03-25T17:58:04.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.118+0000 7fe7e1977640 10 monclient: shutdown 2024-03-25T17:58:04.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.118+0000 7fe7e1977640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:58:04.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.118+0000 7fe7e1977640 1 -- 172.21.15.18:0/3048362600 >> v1:172.21.15.18:6789/0 conn(0x7fe7dc0a98f0 legacy=0x7fe7dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:04.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.118+0000 7fe7e1977640 1 -- 172.21.15.18:0/3048362600 shutdown_connections 2024-03-25T17:58:04.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.118+0000 7fe7e1977640 1 -- 172.21.15.18:0/3048362600 >> 172.21.15.18:0/3048362600 conn(0x7fe7dc0b7ea0 msgr2=0x7fe7dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:58:04.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.118+0000 7fe7e1977640 1 -- 172.21.15.18:0/3048362600 shutdown_connections 2024-03-25T17:58:04.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.118+0000 7fe7e1977640 1 -- 172.21.15.18:0/3048362600 wait complete. 2024-03-25T17:58:04.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.119+0000 7fe7e1977640 10 monclient: build_initial_monmap 2024-03-25T17:58:04.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.119+0000 7fe7e1977640 10 monclient: monmap: 2024-03-25T17:58:04.122 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:58:04.122 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:58:04.122 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:58:04.122 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:58:04.122 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:58:04.122 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:58:04.122 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:58:04.122 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:58:04.122 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:58:04.123 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:04.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.119+0000 7fe7e1977640 1 librados: starting msgr at 2024-03-25T17:58:04.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.119+0000 7fe7e1977640 1 librados: starting objecter 2024-03-25T17:58:04.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.119+0000 7fe7e1977640 1 Processor -- start 2024-03-25T17:58:04.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.120+0000 7fe7e1977640 1 -- start start 2024-03-25T17:58:04.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.120+0000 7fe7e1977640 1 librados: setting wanted keys 2024-03-25T17:58:04.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.120+0000 7fe7e1977640 1 librados: calling monclient init 2024-03-25T17:58:04.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.120+0000 7fe7e1977640 10 monclient: init 2024-03-25T17:58:04.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.120+0000 7fe7e1977640 10 monclient: _reopen_session rank -1 2024-03-25T17:58:04.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.120+0000 7fe7e1977640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:58:04.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.120+0000 7fe7e1977640 10 monclient(hunting): picked mon.noname-c con 0x7fe7dc0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T17:58:04.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.120+0000 7fe7e1977640 10 monclient(hunting): picked mon.noname-a con 0x7fe7dc13f710 addr v1:172.21.15.18:6789/0 2024-03-25T17:58:04.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.120+0000 7fe7e1977640 10 monclient(hunting): picked mon.noname-b con 0x7fe7dc1430c0 addr v1:172.21.15.169:6789/0 2024-03-25T17:58:04.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.120+0000 7fe7e1977640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7dc132a30 con 0x7fe7dc13f710 2024-03-25T17:58:04.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.120+0000 7fe7e1977640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7dc138c80 con 0x7fe7dc1430c0 2024-03-25T17:58:04.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.120+0000 7fe7e1977640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7dc138f60 con 0x7fe7dc0a98f0 2024-03-25T17:58:04.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.120+0000 7fe7e1977640 10 monclient(hunting): _renew_subs 2024-03-25T17:58:04.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.120+0000 7fe7e1977640 10 monclient(hunting): authenticate will time out at 7012.699559s 2024-03-25T17:58:04.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.133+0000 7fe7db7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fe7dc1430c0 0x7fe7dc132320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:50890/0 (socket says 172.21.15.18:50890) 2024-03-25T17:58:04.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.133+0000 7fe7db7fe640 1 -- 172.21.15.18:0/2590314892 learned_addr learned my addr 172.21.15.18:0/2590314892 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:58:04.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.134+0000 7fe7c3fff640 10 client.?.objecter ms_handle_connect 0x7fe7dc1430c0 2024-03-25T17:58:04.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.134+0000 7fe7c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:04.142 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.142+0000 7fe7c3fff640 10 client.?.objecter ms_handle_connect 0x7fe7dc0a98f0 2024-03-25T17:58:04.142 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.142+0000 7fe7c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:04.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.143+0000 7fe7c3fff640 10 client.?.objecter ms_handle_connect 0x7fe7dc13f710 2024-03-25T17:58:04.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.143+0000 7fe7c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:04.150 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.149+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943945001 0 0) 0x7fe7dc138c80 con 0x7fe7dc1430c0 2024-03-25T17:58:04.150 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.149+0000 7fe7c3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:58:04.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.149+0000 7fe7c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:04.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.149+0000 7fe7c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:04.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.149+0000 7fe7c3fff640 10 monclient(hunting): my global_id is 4291 2024-03-25T17:58:04.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.149+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c8004eb0 con 0x7fe7dc1430c0 2024-03-25T17:58:04.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.149+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2616065079 0 0) 0x7fe7dc132a30 con 0x7fe7dc13f710 2024-03-25T17:58:04.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.149+0000 7fe7c3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:58:04.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.149+0000 7fe7c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:04.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.149+0000 7fe7c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:04.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.149+0000 7fe7c3fff640 10 monclient(hunting): my global_id is 4299 2024-03-25T17:58:04.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.149+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7dc138c80 con 0x7fe7dc13f710 2024-03-25T17:58:04.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.150+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172244635 0 0) 0x7fe7dc138f60 con 0x7fe7dc0a98f0 2024-03-25T17:58:04.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.150+0000 7fe7c3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:58:04.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.150+0000 7fe7c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:04.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.150+0000 7fe7c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:04.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.150+0000 7fe7c3fff640 10 monclient(hunting): my global_id is 4307 2024-03-25T17:58:04.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.150+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7dc132a30 con 0x7fe7dc0a98f0 2024-03-25T17:58:04.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.171+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3463883942 0 0) 0x7fe7c8004eb0 con 0x7fe7dc1430c0 2024-03-25T17:58:04.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.171+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7dc138f60 con 0x7fe7dc1430c0 2024-03-25T17:58:04.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.171+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fe7d4001c90 con 0x7fe7dc1430c0 2024-03-25T17:58:04.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.171+0000 7fe7c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:04.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.171+0000 7fe7c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:58:04.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.171+0000 7fe7c3fff640 10 monclient(hunting): dump: 2024-03-25T17:58:04.172 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:04.172 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:04.172 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:04.172 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:04.172 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:04.172 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:04.173 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:04.173 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:04.173 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:04.173 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:04.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.171+0000 7fe7c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:58:04.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.172+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 836510861 0 0) 0x7fe7dc138c80 con 0x7fe7dc13f710 2024-03-25T17:58:04.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.172+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7c8004eb0 con 0x7fe7dc13f710 2024-03-25T17:58:04.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.172+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fe7d0001930 con 0x7fe7dc13f710 2024-03-25T17:58:04.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.172+0000 7fe7c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:04.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.172+0000 7fe7c3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:58:04.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.172+0000 7fe7c3fff640 10 monclient(hunting): dump: 2024-03-25T17:58:04.174 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:04.174 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:04.174 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:04.174 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:04.174 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:04.174 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:04.174 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:04.175 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:04.175 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:04.175 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:04.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.176+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 378149334 0 0) 0x7fe7dc132a30 con 0x7fe7dc0a98f0 2024-03-25T17:58:04.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.177+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7dc138c80 con 0x7fe7dc0a98f0 2024-03-25T17:58:04.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.177+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fe7cc001b00 con 0x7fe7dc0a98f0 2024-03-25T17:58:04.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.177+0000 7fe7c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:04.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.177+0000 7fe7c3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:58:04.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.177+0000 7fe7c3fff640 10 monclient(hunting): dump: 2024-03-25T17:58:04.178 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:04.178 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:04.178 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:04.178 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:04.178 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:04.178 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:04.178 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:04.178 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:04.178 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:04.178 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:04.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 64254511 0 0) 0x7fe7dc138f60 con 0x7fe7dc1430c0 2024-03-25T17:58:04.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 >> v1:172.21.15.18:6790/0 conn(0x7fe7dc0a98f0 legacy=0x7fe7dc137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:04.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 >> v1:172.21.15.18:6789/0 conn(0x7fe7dc13f710 legacy=0x7fe7dc131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:04.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7c3fff640 10 monclient: _finish_hunting 0 2024-03-25T17:58:04.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7c3fff640 1 monclient: found mon.b 2024-03-25T17:58:04.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:58:04.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7c3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:58:04.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7dc139240 con 0x7fe7dc1430c0 2024-03-25T17:58:04.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7c3fff640 10 monclient: _finish_auth 0 2024-03-25T17:58:04.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7c3fff640 10 monclient: _check_auth_tickets 2024-03-25T17:58:04.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:58:04.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7e1977640 5 monclient: authenticate success, global_id 4291 2024-03-25T17:58:04.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7e1977640 10 monclient: _renew_subs 2024-03-25T17:58:04.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7e1977640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:58:04.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7e1977640 1 -- 172.21.15.18:0/2590314892 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7dc136e50 con 0x7fe7dc1430c0 2024-03-25T17:58:04.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7e1977640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:58:04.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7e1977640 10 monclient: _renew_subs 2024-03-25T17:58:04.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7e1977640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:58:04.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7e1977640 1 -- 172.21.15.18:0/2590314892 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7dc1389f0 con 0x7fe7dc1430c0 2024-03-25T17:58:04.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.180+0000 7fe7e1977640 1 librados: init done 2024-03-25T17:58:04.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.181+0000 7fe7e1977640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:04.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.181+0000 7fe7e1977640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:04.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.181+0000 7fe7e1977640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:58:04.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.181+0000 7fe7e1977640 1 -- 172.21.15.18:0/2590314892 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7dc0627b0 con 0x7fe7dc1430c0 2024-03-25T17:58:04.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.457+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7d4001df0 con 0x7fe7dc1430c0 2024-03-25T17:58:04.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.457+0000 7fe7c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:58:04.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.457+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fe7d4003b70 con 0x7fe7dc1430c0 2024-03-25T17:58:04.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.457+0000 7fe7c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:04.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.457+0000 7fe7c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:58:04.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.457+0000 7fe7c3fff640 10 monclient: dump: 2024-03-25T17:58:04.458 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:04.458 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:04.458 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:04.458 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:04.459 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:04.459 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:04.459 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:04.459 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:04.459 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:04.459 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:04.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.457+0000 7fe7d97fa640 4 set_mon_vals no callback set 2024-03-25T17:58:04.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.458+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fe7d4003d90 con 0x7fe7dc1430c0 2024-03-25T17:58:04.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.458+0000 7fe7daffd640 10 monclient: get_auth_request con 0x7fe7c806e110 auth_method 0 2024-03-25T17:58:04.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.459+0000 7fe7c3fff640 10 client.4291.objecter ms_handle_connect 0x7fe7c806e110 2024-03-25T17:58:04.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.463+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fe7d4054b30 con 0x7fe7dc1430c0 2024-03-25T17:58:04.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.463+0000 7fe7c3fff640 10 client.4291.objecter ms_dispatch 0x7fe7dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:58:04.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.463+0000 7fe7c3fff640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:58:04.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.463+0000 7fe7c3fff640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:58:04.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.463+0000 7fe7c3fff640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-25T17:58:04.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.463+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7d4050470 con 0x7fe7dc1430c0 2024-03-25T17:58:04.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.463+0000 7fe7c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:04.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.463+0000 7fe7c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:58:04.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.724+0000 7fe7e1977640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:04.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.724+0000 7fe7e1977640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:04.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.724+0000 7fe7e1977640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:58:04.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.724+0000 7fe7e1977640 1 -- 172.21.15.18:0/2590314892 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7dc13dd60 con 0x7fe7dc1430c0 2024-03-25T17:58:04.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.766+0000 7fe7c3fff640 1 -- 172.21.15.18:0/2590314892 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fe7d4052570 con 0x7fe7dc1430c0 2024-03-25T17:58:04.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.766+0000 7fe7c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:04.768 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.766+0000 7fe7c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:58:04.768 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:58:04.768 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:58:04.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.769+0000 7fe7e1977640 10 librados: watch_flush enter 2024-03-25T17:58:04.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.769+0000 7fe7e1977640 10 librados: watch_flush exit 2024-03-25T17:58:04.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.769+0000 7fe7e1977640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-25T17:58:04.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.769+0000 7fe7e1977640 10 client.4291.objecter successfully canceled tick 2024-03-25T17:58:04.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.769+0000 7fe7e1977640 1 -- 172.21.15.18:0/2590314892 >> v1:172.21.15.18:6800/3721244451 conn(0x7fe7c806e110 legacy=0x7fe7c80705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:04.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.769+0000 7fe7e1977640 10 monclient: shutdown 2024-03-25T17:58:04.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.769+0000 7fe7e1977640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:58:04.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.769+0000 7fe7e1977640 1 -- 172.21.15.18:0/2590314892 >> v1:172.21.15.169:6789/0 conn(0x7fe7dc1430c0 legacy=0x7fe7dc132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:04.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.769+0000 7fe7e1977640 1 -- 172.21.15.18:0/2590314892 shutdown_connections 2024-03-25T17:58:04.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.769+0000 7fe7e1977640 1 -- 172.21.15.18:0/2590314892 >> 172.21.15.18:0/2590314892 conn(0x7fe7dc0b7ea0 msgr2=0x7fe7dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:58:04.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.769+0000 7fe7e1977640 1 -- 172.21.15.18:0/2590314892 shutdown_connections 2024-03-25T17:58:04.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.769+0000 7fe7e1977640 1 -- 172.21.15.18:0/2590314892 wait complete. 2024-03-25T17:58:04.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:04.769+0000 7fe7e1977640 1 librados: shutdown 2024-03-25T17:58:04.781 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:58:11.587 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:58:11.699 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.699+0000 7f9caad8c640 10 monclient: get_monmap_and_config 2024-03-25T17:58:11.699 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.699+0000 7f9caad8c640 10 monclient: build_initial_monmap 2024-03-25T17:58:11.699 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.699+0000 7f9caad8c640 10 monclient: monmap: 2024-03-25T17:58:11.699 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:58:11.699 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:11.699 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:58:11.699888+0000 2024-03-25T17:58:11.699 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:58:11.699888+0000 2024-03-25T17:58:11.699 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:58:11.700 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:11.700 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:58:11.700 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:58:11.700 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:58:11.700 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:11.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.700+0000 7f9caad8c640 1 Processor -- start 2024-03-25T17:58:11.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.700+0000 7f9caad8c640 1 -- start start 2024-03-25T17:58:11.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.700+0000 7f9caad8c640 10 monclient: init 2024-03-25T17:58:11.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.701+0000 7f9caad8c640 10 monclient: _reopen_session rank -1 2024-03-25T17:58:11.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.701+0000 7f9caad8c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:58:11.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.701+0000 7f9caad8c640 10 monclient(hunting): picked mon.noname-b con 0x7f9ca40a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T17:58:11.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.701+0000 7f9caad8c640 10 monclient(hunting): picked mon.noname-a con 0x7f9ca413f710 addr v1:172.21.15.18:6789/0 2024-03-25T17:58:11.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.701+0000 7f9caad8c640 10 monclient(hunting): picked mon.noname-c con 0x7f9ca41430c0 addr v1:172.21.15.169:6789/0 2024-03-25T17:58:11.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.701+0000 7f9caad8c640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca4142520 con 0x7f9ca413f710 2024-03-25T17:58:11.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.701+0000 7f9caad8c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca40b28a0 con 0x7f9ca41430c0 2024-03-25T17:58:11.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.701+0000 7f9caad8c640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca40b2a30 con 0x7f9ca40a98f0 2024-03-25T17:58:11.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.701+0000 7f9caad8c640 10 monclient(hunting): _renew_subs 2024-03-25T17:58:11.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.701+0000 7f9caad8c640 10 monclient(hunting): authenticate will time out at 7020.279051s 2024-03-25T17:58:11.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.707+0000 7f9ca8b28640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f9ca40a98f0 0x7f9ca40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:41146/0 (socket says 172.21.15.18:41146) 2024-03-25T17:58:11.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.707+0000 7f9ca8b28640 1 -- 172.21.15.18:0/1626715337 learned_addr learned my addr 172.21.15.18:0/1626715337 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:58:11.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.721+0000 7f9ca37fe640 1 -- 172.21.15.18:0/1626715337 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544569127 0 0) 0x7f9ca40b2a30 con 0x7f9ca40a98f0 2024-03-25T17:58:11.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.721+0000 7f9ca37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:58:11.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.721+0000 7f9ca37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:11.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.721+0000 7f9ca37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:11.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.721+0000 7f9ca37fe640 10 monclient(hunting): my global_id is 4310 2024-03-25T17:58:11.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.721+0000 7f9ca37fe640 1 -- 172.21.15.18:0/1626715337 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c94003520 con 0x7f9ca40a98f0 2024-03-25T17:58:11.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.731+0000 7f9ca37fe640 1 -- 172.21.15.18:0/1626715337 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560685796 0 0) 0x7f9ca4142520 con 0x7f9ca413f710 2024-03-25T17:58:11.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.731+0000 7f9ca37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:58:11.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.731+0000 7f9ca37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:11.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.731+0000 7f9ca37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:11.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.731+0000 7f9ca37fe640 10 monclient(hunting): my global_id is 4302 2024-03-25T17:58:11.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.731+0000 7f9ca37fe640 1 -- 172.21.15.18:0/1626715337 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ca40b2a30 con 0x7f9ca413f710 2024-03-25T17:58:11.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.741+0000 7f9ca37fe640 1 -- 172.21.15.18:0/1626715337 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890007993 0 0) 0x7f9ca40b28a0 con 0x7f9ca41430c0 2024-03-25T17:58:11.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.741+0000 7f9ca37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:58:11.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.741+0000 7f9ca37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:11.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.741+0000 7f9ca37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:11.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.741+0000 7f9ca37fe640 10 monclient(hunting): my global_id is 4294 2024-03-25T17:58:11.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.741+0000 7f9ca37fe640 1 -- 172.21.15.18:0/1626715337 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ca4142520 con 0x7f9ca41430c0 2024-03-25T17:58:11.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9ca37fe640 1 -- 172.21.15.18:0/1626715337 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 533109148 0 0) 0x7f9ca40b2a30 con 0x7f9ca413f710 2024-03-25T17:58:11.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9ca37fe640 1 -- 172.21.15.18:0/1626715337 >> v1:172.21.15.18:6790/0 conn(0x7f9ca40a98f0 legacy=0x7f9ca40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:11.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9ca37fe640 1 -- 172.21.15.18:0/1626715337 >> v1:172.21.15.169:6789/0 conn(0x7f9ca41430c0 legacy=0x7f9ca414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:11.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9ca37fe640 10 monclient: _finish_hunting 0 2024-03-25T17:58:11.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9ca37fe640 1 monclient: found mon.noname-a 2024-03-25T17:58:11.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9ca37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:58:11.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9ca37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T17:58:11.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9ca37fe640 1 -- 172.21.15.18:0/1626715337 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca40b44c0 con 0x7f9ca413f710 2024-03-25T17:58:11.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9ca37fe640 10 monclient: _finish_auth 0 2024-03-25T17:58:11.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9ca37fe640 10 monclient: _check_auth_tickets 2024-03-25T17:58:11.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9ca37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:58:11.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9ca37fe640 1 -- 172.21.15.18:0/1626715337 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9c90001670 con 0x7f9ca413f710 2024-03-25T17:58:11.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9caad8c640 5 monclient: authenticate success, global_id 4302 2024-03-25T17:58:11.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9caad8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:58:11.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9ca37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:11.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9ca37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:58:11.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9ca37fe640 10 monclient: dump: 2024-03-25T17:58:11.747 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:11.747 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:11.747 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:11.747 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:11.747 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:11.747 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:11.748 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:11.748 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:11.748 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:11.748 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:11.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.746+0000 7f9caad8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:58:11.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.756+0000 7f9ca37fe640 1 -- 172.21.15.18:0/1626715337 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c900017d0 con 0x7f9ca413f710 2024-03-25T17:58:11.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.756+0000 7f9ca37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:58:11.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.756+0000 7f9ca37fe640 1 -- 172.21.15.18:0/1626715337 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9c90001be0 con 0x7f9ca413f710 2024-03-25T17:58:11.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.756+0000 7f9ca37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:11.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.756+0000 7f9ca37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:58:11.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.756+0000 7f9ca37fe640 10 monclient: dump: 2024-03-25T17:58:11.756 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:11.756 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:11.756 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:11.757 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:11.757 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:11.757 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:11.757 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:11.757 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:11.757 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:11.757 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:11.757 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.756+0000 7f9caad8c640 10 monclient: get_monmap_and_config success 2024-03-25T17:58:11.757 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.756+0000 7f9caad8c640 4 set_mon_vals no callback set 2024-03-25T17:58:11.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.757+0000 7f9caad8c640 10 monclient: shutdown 2024-03-25T17:58:11.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.757+0000 7f9caad8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:58:11.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.757+0000 7f9caad8c640 1 -- 172.21.15.18:0/1626715337 >> v1:172.21.15.18:6789/0 conn(0x7f9ca413f710 legacy=0x7f9ca41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:11.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.757+0000 7f9caad8c640 1 -- 172.21.15.18:0/1626715337 shutdown_connections 2024-03-25T17:58:11.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.757+0000 7f9caad8c640 1 -- 172.21.15.18:0/1626715337 >> 172.21.15.18:0/1626715337 conn(0x7f9ca40b7ea0 msgr2=0x7f9ca40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:58:11.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.757+0000 7f9caad8c640 1 -- 172.21.15.18:0/1626715337 shutdown_connections 2024-03-25T17:58:11.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.757+0000 7f9caad8c640 1 -- 172.21.15.18:0/1626715337 wait complete. 2024-03-25T17:58:11.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.758+0000 7f9caad8c640 10 monclient: build_initial_monmap 2024-03-25T17:58:11.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.758+0000 7f9caad8c640 10 monclient: monmap: 2024-03-25T17:58:11.760 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:58:11.760 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:58:11.760 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:58:11.760 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:58:11.760 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:58:11.760 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:58:11.760 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:58:11.760 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:58:11.760 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:58:11.761 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:11.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.758+0000 7f9caad8c640 1 librados: starting msgr at 2024-03-25T17:58:11.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.758+0000 7f9caad8c640 1 librados: starting objecter 2024-03-25T17:58:11.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.759+0000 7f9caad8c640 1 Processor -- start 2024-03-25T17:58:11.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.759+0000 7f9caad8c640 1 -- start start 2024-03-25T17:58:11.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.759+0000 7f9caad8c640 1 librados: setting wanted keys 2024-03-25T17:58:11.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.759+0000 7f9caad8c640 1 librados: calling monclient init 2024-03-25T17:58:11.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.759+0000 7f9caad8c640 10 monclient: init 2024-03-25T17:58:11.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.759+0000 7f9caad8c640 10 monclient: _reopen_session rank -1 2024-03-25T17:58:11.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.759+0000 7f9caad8c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:58:11.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.759+0000 7f9caad8c640 10 monclient(hunting): picked mon.noname-b con 0x7f9ca40a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T17:58:11.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.759+0000 7f9caad8c640 10 monclient(hunting): picked mon.noname-a con 0x7f9ca413f710 addr v1:172.21.15.18:6789/0 2024-03-25T17:58:11.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.759+0000 7f9caad8c640 10 monclient(hunting): picked mon.noname-c con 0x7f9ca41430c0 addr v1:172.21.15.18:6790/0 2024-03-25T17:58:11.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.759+0000 7f9caad8c640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca413c530 con 0x7f9ca413f710 2024-03-25T17:58:11.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.759+0000 7f9caad8c640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca412f860 con 0x7f9ca40a98f0 2024-03-25T17:58:11.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.759+0000 7f9caad8c640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca412fad0 con 0x7f9ca41430c0 2024-03-25T17:58:11.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.760+0000 7f9caad8c640 10 monclient(hunting): _renew_subs 2024-03-25T17:58:11.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.760+0000 7f9caad8c640 10 monclient(hunting): authenticate will time out at 7020.337662s 2024-03-25T17:58:11.774 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.774+0000 7f9ca9329640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f9ca41430c0 0x7f9ca41e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:41192/0 (socket says 172.21.15.18:41192) 2024-03-25T17:58:11.774 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.774+0000 7f9ca8b28640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f9ca40a98f0 0x7f9ca40b7590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:56132/0 (socket says 172.21.15.18:56132) 2024-03-25T17:58:11.774 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.774+0000 7f9ca9329640 1 -- 172.21.15.18:0/4293694331 learned_addr learned my addr 172.21.15.18:0/4293694331 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:58:11.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.776+0000 7f9ca17fa640 10 client.?.objecter ms_handle_connect 0x7f9ca41430c0 2024-03-25T17:58:11.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.776+0000 7f9ca17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:11.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.777+0000 7f9ca17fa640 10 client.?.objecter ms_handle_connect 0x7f9ca413f710 2024-03-25T17:58:11.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.777+0000 7f9ca17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:11.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.778+0000 7f9ca17fa640 10 client.?.objecter ms_handle_connect 0x7f9ca40a98f0 2024-03-25T17:58:11.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.778+0000 7f9ca17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:11.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.787+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899181589 0 0) 0x7f9ca412fad0 con 0x7f9ca41430c0 2024-03-25T17:58:11.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.787+0000 7f9ca17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:58:11.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.787+0000 7f9ca17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:11.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.787+0000 7f9ca17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:11.787 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.787+0000 7f9ca17fa640 10 monclient(hunting): my global_id is 4313 2024-03-25T17:58:11.787 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.787+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c88004eb0 con 0x7f9ca41430c0 2024-03-25T17:58:11.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.788+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127016181 0 0) 0x7f9ca413c530 con 0x7f9ca413f710 2024-03-25T17:58:11.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.788+0000 7f9ca17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:58:11.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.788+0000 7f9ca17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:11.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.788+0000 7f9ca17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:11.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.788+0000 7f9ca17fa640 10 monclient(hunting): my global_id is 4305 2024-03-25T17:58:11.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.788+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ca412fad0 con 0x7f9ca413f710 2024-03-25T17:58:11.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.790+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484765790 0 0) 0x7f9ca412f860 con 0x7f9ca40a98f0 2024-03-25T17:58:11.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.790+0000 7f9ca17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:58:11.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.790+0000 7f9ca17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:11.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.790+0000 7f9ca17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:11.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.790+0000 7f9ca17fa640 10 monclient(hunting): my global_id is 4297 2024-03-25T17:58:11.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.790+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ca413c530 con 0x7f9ca40a98f0 2024-03-25T17:58:11.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.794+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2480216889 0 0) 0x7f9ca412fad0 con 0x7f9ca413f710 2024-03-25T17:58:11.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.794+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ca412f860 con 0x7f9ca413f710 2024-03-25T17:58:11.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.796+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9c90001b60 con 0x7f9ca413f710 2024-03-25T17:58:11.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.796+0000 7f9ca17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:11.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.796+0000 7f9ca17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:58:11.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.796+0000 7f9ca17fa640 10 monclient(hunting): dump: 2024-03-25T17:58:11.796 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:11.796 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:11.796 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:11.796 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:11.796 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:11.796 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:11.797 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:11.797 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:11.797 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:11.797 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:11.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.796+0000 7f9ca17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:58:11.798 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.799+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3505096132 0 0) 0x7f9c88004eb0 con 0x7f9ca41430c0 2024-03-25T17:58:11.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.799+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ca412fad0 con 0x7f9ca41430c0 2024-03-25T17:58:11.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.799+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9c9c001c70 con 0x7f9ca41430c0 2024-03-25T17:58:11.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.799+0000 7f9ca17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:11.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.799+0000 7f9ca17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:58:11.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.799+0000 7f9ca17fa640 10 monclient(hunting): dump: 2024-03-25T17:58:11.799 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:11.799 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:11.799 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:11.799 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:11.799 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:11.799 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:11.799 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:11.799 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:11.799 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:11.800 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:11.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1557489784 0 0) 0x7f9ca412f860 con 0x7f9ca413f710 2024-03-25T17:58:11.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 >> v1:172.21.15.18:6790/0 conn(0x7f9ca41430c0 legacy=0x7f9ca41e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:11.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 >> v1:172.21.15.169:6789/0 conn(0x7f9ca40a98f0 legacy=0x7f9ca40b7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:11.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9ca17fa640 10 monclient: _finish_hunting 0 2024-03-25T17:58:11.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9ca17fa640 1 monclient: found mon.a 2024-03-25T17:58:11.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9ca17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:58:11.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9ca17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:58:11.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca4131320 con 0x7f9ca413f710 2024-03-25T17:58:11.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9ca17fa640 10 monclient: _finish_auth 0 2024-03-25T17:58:11.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9ca17fa640 10 monclient: _check_auth_tickets 2024-03-25T17:58:11.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9ca17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:58:11.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9caad8c640 5 monclient: authenticate success, global_id 4305 2024-03-25T17:58:11.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9caad8c640 10 monclient: _renew_subs 2024-03-25T17:58:11.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9caad8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:58:11.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9caad8c640 1 -- 172.21.15.18:0/4293694331 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca413c860 con 0x7f9ca413f710 2024-03-25T17:58:11.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9caad8c640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:58:11.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9caad8c640 10 monclient: _renew_subs 2024-03-25T17:58:11.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9caad8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:58:11.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9caad8c640 1 -- 172.21.15.18:0/4293694331 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca413bf20 con 0x7f9ca413f710 2024-03-25T17:58:11.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.802+0000 7f9caad8c640 1 librados: init done 2024-03-25T17:58:11.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.803+0000 7f9caad8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:11.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.803+0000 7f9caad8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:11.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.803+0000 7f9caad8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:58:11.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:11.803+0000 7f9caad8c640 1 -- 172.21.15.18:0/4293694331 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca40627b0 con 0x7f9ca413f710 2024-03-25T17:58:12.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.121+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c900038a0 con 0x7f9ca413f710 2024-03-25T17:58:12.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.121+0000 7f9ca17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:58:12.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.121+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9c90003bb0 con 0x7f9ca413f710 2024-03-25T17:58:12.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.121+0000 7f9ca17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:12.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.121+0000 7f9ca17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:58:12.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.121+0000 7f9ca17fa640 10 monclient: dump: 2024-03-25T17:58:12.122 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:12.122 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:12.122 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:12.122 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:12.122 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:12.122 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:12.122 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:12.122 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:12.122 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:12.122 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:12.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.121+0000 7f9ca2ffd640 4 set_mon_vals no callback set 2024-03-25T17:58:12.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.122+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f9c90019b80 con 0x7f9ca413f710 2024-03-25T17:58:12.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.123+0000 7f9ca8b28640 10 monclient: get_auth_request con 0x7f9c8806df60 auth_method 0 2024-03-25T17:58:12.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.123+0000 7f9ca17fa640 10 client.4305.objecter ms_handle_connect 0x7f9c8806df60 2024-03-25T17:58:12.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.132+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f9c900536b0 con 0x7f9ca413f710 2024-03-25T17:58:12.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.132+0000 7f9ca17fa640 10 client.4305.objecter ms_dispatch 0x7f9ca4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:58:12.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.132+0000 7f9ca17fa640 3 client.4305.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:58:12.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.132+0000 7f9ca17fa640 3 client.4305.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:58:12.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.132+0000 7f9ca17fa640 20 client.4305.objecter dump_active .. 0 homeless 2024-03-25T17:58:12.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.132+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c9004f1c0 con 0x7f9ca413f710 2024-03-25T17:58:12.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.132+0000 7f9ca17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:12.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.132+0000 7f9ca17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:58:12.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.391+0000 7f9caad8c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:12.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.391+0000 7f9caad8c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:12.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.391+0000 7f9caad8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:58:12.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.391+0000 7f9caad8c640 1 -- 172.21.15.18:0/4293694331 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ca4074cc0 con 0x7f9ca413f710 2024-03-25T17:58:12.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.439+0000 7f9ca17fa640 1 -- 172.21.15.18:0/4293694331 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f9c900510f0 con 0x7f9ca413f710 2024-03-25T17:58:12.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.439+0000 7f9ca17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:12.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.439+0000 7f9ca17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:58:12.439 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:58:12.439 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:58:12.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.441+0000 7f9caad8c640 10 librados: watch_flush enter 2024-03-25T17:58:12.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.441+0000 7f9caad8c640 10 librados: watch_flush exit 2024-03-25T17:58:12.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.441+0000 7f9caad8c640 20 client.4305.objecter shutdown clearing up homeless session... 2024-03-25T17:58:12.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.441+0000 7f9caad8c640 10 client.4305.objecter successfully canceled tick 2024-03-25T17:58:12.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.441+0000 7f9caad8c640 1 -- 172.21.15.18:0/4293694331 >> v1:172.21.15.18:6800/3721244451 conn(0x7f9c8806df60 legacy=0x7f9c88070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:12.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.441+0000 7f9caad8c640 10 monclient: shutdown 2024-03-25T17:58:12.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.441+0000 7f9caad8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:58:12.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.441+0000 7f9caad8c640 1 -- 172.21.15.18:0/4293694331 >> v1:172.21.15.18:6789/0 conn(0x7f9ca413f710 legacy=0x7f9ca41df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:12.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.442+0000 7f9caad8c640 1 -- 172.21.15.18:0/4293694331 shutdown_connections 2024-03-25T17:58:12.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.442+0000 7f9caad8c640 1 -- 172.21.15.18:0/4293694331 >> 172.21.15.18:0/4293694331 conn(0x7f9ca40b7ea0 msgr2=0x7f9ca4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:58:12.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.442+0000 7f9caad8c640 1 -- 172.21.15.18:0/4293694331 shutdown_connections 2024-03-25T17:58:12.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.442+0000 7f9caad8c640 1 -- 172.21.15.18:0/4293694331 wait complete. 2024-03-25T17:58:12.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:12.442+0000 7f9caad8c640 1 librados: shutdown 2024-03-25T17:58:12.453 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:58:19.260 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:58:19.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.374+0000 7f863530e640 10 monclient: get_monmap_and_config 2024-03-25T17:58:19.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.374+0000 7f863530e640 10 monclient: build_initial_monmap 2024-03-25T17:58:19.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.374+0000 7f863530e640 10 monclient: monmap: 2024-03-25T17:58:19.374 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:58:19.374 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:19.375 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:58:19.375136+0000 2024-03-25T17:58:19.375 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:58:19.375136+0000 2024-03-25T17:58:19.375 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:58:19.375 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:19.375 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:58:19.375 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:58:19.375 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:58:19.375 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:19.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.375+0000 7f863530e640 1 Processor -- start 2024-03-25T17:58:19.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.375+0000 7f863530e640 1 -- start start 2024-03-25T17:58:19.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.375+0000 7f863530e640 10 monclient: init 2024-03-25T17:58:19.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.376+0000 7f863530e640 10 monclient: _reopen_session rank -1 2024-03-25T17:58:19.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.376+0000 7f863530e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:58:19.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.376+0000 7f863530e640 10 monclient(hunting): picked mon.noname-b con 0x7f86300a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T17:58:19.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.376+0000 7f863530e640 10 monclient(hunting): picked mon.noname-c con 0x7f863013f710 addr v1:172.21.15.169:6789/0 2024-03-25T17:58:19.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.376+0000 7f863530e640 10 monclient(hunting): picked mon.noname-a con 0x7f86301430c0 addr v1:172.21.15.18:6789/0 2024-03-25T17:58:19.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.376+0000 7f863530e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8630142520 con 0x7f86301430c0 2024-03-25T17:58:19.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.376+0000 7f863530e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86300b28a0 con 0x7f863013f710 2024-03-25T17:58:19.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.376+0000 7f863530e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86300b2a30 con 0x7f86300a98f0 2024-03-25T17:58:19.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.376+0000 7f863530e640 10 monclient(hunting): _renew_subs 2024-03-25T17:58:19.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.376+0000 7f863530e640 10 monclient(hunting): authenticate will time out at 7027.953972s 2024-03-25T17:58:19.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.387+0000 7f862effd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f86300a98f0 0x7f86300a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:51620/0 (socket says 172.21.15.18:51620) 2024-03-25T17:58:19.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.387+0000 7f862effd640 1 -- 172.21.15.18:0/1119479954 learned_addr learned my addr 172.21.15.18:0/1119479954 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:58:19.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.399+0000 7f862dffb640 1 -- 172.21.15.18:0/1119479954 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561008594 0 0) 0x7f86300b2a30 con 0x7f86300a98f0 2024-03-25T17:58:19.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.399+0000 7f862dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:58:19.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.399+0000 7f862dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:19.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.399+0000 7f862dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:19.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.399+0000 7f862dffb640 10 monclient(hunting): my global_id is 4316 2024-03-25T17:58:19.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.399+0000 7f862dffb640 1 -- 172.21.15.18:0/1119479954 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8618003520 con 0x7f86300a98f0 2024-03-25T17:58:19.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.420+0000 7f862dffb640 1 -- 172.21.15.18:0/1119479954 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474297274 0 0) 0x7f86300b28a0 con 0x7f863013f710 2024-03-25T17:58:19.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.420+0000 7f862dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:58:19.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.420+0000 7f862dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:19.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.420+0000 7f862dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:19.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.420+0000 7f862dffb640 10 monclient(hunting): my global_id is 4300 2024-03-25T17:58:19.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.420+0000 7f862dffb640 1 -- 172.21.15.18:0/1119479954 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86300b2a30 con 0x7f863013f710 2024-03-25T17:58:19.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f862dffb640 1 -- 172.21.15.18:0/1119479954 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3850350785 0 0) 0x7f8618003520 con 0x7f86300a98f0 2024-03-25T17:58:19.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f862dffb640 1 -- 172.21.15.18:0/1119479954 >> v1:172.21.15.169:6789/0 conn(0x7f863013f710 legacy=0x7f86301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:19.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f862dffb640 1 -- 172.21.15.18:0/1119479954 >> v1:172.21.15.18:6789/0 conn(0x7f86301430c0 legacy=0x7f863014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:19.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f862dffb640 10 monclient: _finish_hunting 0 2024-03-25T17:58:19.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f862dffb640 1 monclient: found mon.noname-b 2024-03-25T17:58:19.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f862dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:58:19.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f862dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T17:58:19.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f862dffb640 1 -- 172.21.15.18:0/1119479954 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86300b44c0 con 0x7f86300a98f0 2024-03-25T17:58:19.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f862dffb640 10 monclient: _finish_auth 0 2024-03-25T17:58:19.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f862dffb640 10 monclient: _check_auth_tickets 2024-03-25T17:58:19.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f862dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:58:19.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f862dffb640 1 -- 172.21.15.18:0/1119479954 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8620001690 con 0x7f86300a98f0 2024-03-25T17:58:19.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f863530e640 5 monclient: authenticate success, global_id 4316 2024-03-25T17:58:19.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f863530e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:58:19.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f862dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:19.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f862dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:58:19.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f862dffb640 10 monclient: dump: 2024-03-25T17:58:19.425 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:19.425 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:19.425 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:19.425 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:19.425 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:19.425 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:19.425 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:19.425 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:19.426 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:19.426 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:19.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.424+0000 7f863530e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:58:19.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.436+0000 7f862dffb640 1 -- 172.21.15.18:0/1119479954 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86200017f0 con 0x7f86300a98f0 2024-03-25T17:58:19.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.436+0000 7f862dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:58:19.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.436+0000 7f862dffb640 1 -- 172.21.15.18:0/1119479954 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8620001c00 con 0x7f86300a98f0 2024-03-25T17:58:19.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.436+0000 7f863530e640 10 monclient: get_monmap_and_config success 2024-03-25T17:58:19.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.436+0000 7f863530e640 4 set_mon_vals no callback set 2024-03-25T17:58:19.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.436+0000 7f862dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:19.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.436+0000 7f862dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:58:19.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.436+0000 7f862dffb640 10 monclient: dump: 2024-03-25T17:58:19.436 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:19.436 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:19.436 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:19.436 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:19.436 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:19.436 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:19.436 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:19.437 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:19.437 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:19.437 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:19.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.437+0000 7f863530e640 10 monclient: shutdown 2024-03-25T17:58:19.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.437+0000 7f863530e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:58:19.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.437+0000 7f863530e640 1 -- 172.21.15.18:0/1119479954 >> v1:172.21.15.18:6790/0 conn(0x7f86300a98f0 legacy=0x7f86300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:19.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.437+0000 7f863530e640 1 -- 172.21.15.18:0/1119479954 shutdown_connections 2024-03-25T17:58:19.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.437+0000 7f863530e640 1 -- 172.21.15.18:0/1119479954 >> 172.21.15.18:0/1119479954 conn(0x7f86300b7ea0 msgr2=0x7f86300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:58:19.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.437+0000 7f863530e640 1 -- 172.21.15.18:0/1119479954 shutdown_connections 2024-03-25T17:58:19.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.437+0000 7f863530e640 1 -- 172.21.15.18:0/1119479954 wait complete. 2024-03-25T17:58:19.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.438+0000 7f863530e640 10 monclient: build_initial_monmap 2024-03-25T17:58:19.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.438+0000 7f863530e640 10 monclient: monmap: 2024-03-25T17:58:19.440 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:58:19.440 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:58:19.440 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:58:19.440 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:58:19.440 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:58:19.440 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:58:19.441 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:58:19.441 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:58:19.441 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:58:19.441 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:19.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.438+0000 7f863530e640 1 librados: starting msgr at 2024-03-25T17:58:19.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.438+0000 7f863530e640 1 librados: starting objecter 2024-03-25T17:58:19.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.438+0000 7f863530e640 1 Processor -- start 2024-03-25T17:58:19.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.438+0000 7f863530e640 1 -- start start 2024-03-25T17:58:19.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.438+0000 7f863530e640 1 librados: setting wanted keys 2024-03-25T17:58:19.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.438+0000 7f863530e640 1 librados: calling monclient init 2024-03-25T17:58:19.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.438+0000 7f863530e640 10 monclient: init 2024-03-25T17:58:19.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.439+0000 7f863530e640 10 monclient: _reopen_session rank -1 2024-03-25T17:58:19.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.439+0000 7f863530e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:58:19.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.439+0000 7f863530e640 10 monclient(hunting): picked mon.noname-b con 0x7f86300a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T17:58:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.439+0000 7f863530e640 10 monclient(hunting): picked mon.noname-c con 0x7f863013f710 addr v1:172.21.15.18:6790/0 2024-03-25T17:58:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.439+0000 7f863530e640 10 monclient(hunting): picked mon.noname-a con 0x7f86301430c0 addr v1:172.21.15.18:6789/0 2024-03-25T17:58:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.439+0000 7f863530e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f863013be70 con 0x7f86301430c0 2024-03-25T17:58:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.439+0000 7f863530e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8630131660 con 0x7f86300a98f0 2024-03-25T17:58:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.439+0000 7f863530e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86301317f0 con 0x7f863013f710 2024-03-25T17:58:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.439+0000 7f863530e640 10 monclient(hunting): _renew_subs 2024-03-25T17:58:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.439+0000 7f863530e640 10 monclient(hunting): authenticate will time out at 7028.017409s 2024-03-25T17:58:19.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.448+0000 7f862effd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f86300a98f0 0x7f8630130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:45640/0 (socket says 172.21.15.18:45640) 2024-03-25T17:58:19.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.448+0000 7f862effd640 1 -- 172.21.15.18:0/2289722250 learned_addr learned my addr 172.21.15.18:0/2289722250 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:58:19.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.449+0000 7f86177fe640 10 client.?.objecter ms_handle_connect 0x7f86301430c0 2024-03-25T17:58:19.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.449+0000 7f86177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:19.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.453+0000 7f86177fe640 10 client.?.objecter ms_handle_connect 0x7f863013f710 2024-03-25T17:58:19.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.453+0000 7f86177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:19.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.456+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885839779 0 0) 0x7f863013be70 con 0x7f86301430c0 2024-03-25T17:58:19.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.456+0000 7f86177fe640 10 monclient(hunting): _negotiate 2024-03-25T17:58:19.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.456+0000 7f86177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:19.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.456+0000 7f86177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:19.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.456+0000 7f86177fe640 10 monclient(hunting): my global_id is 4311 2024-03-25T17:58:19.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.456+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f861c004eb0 con 0x7f86301430c0 2024-03-25T17:58:19.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.459+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247720112 0 0) 0x7f86301317f0 con 0x7f863013f710 2024-03-25T17:58:19.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.459+0000 7f86177fe640 10 monclient(hunting): _negotiate 2024-03-25T17:58:19.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.459+0000 7f86177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:19.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.459+0000 7f86177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:19.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.459+0000 7f86177fe640 10 monclient(hunting): my global_id is 4319 2024-03-25T17:58:19.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.459+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f863013be70 con 0x7f863013f710 2024-03-25T17:58:19.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.461+0000 7f86177fe640 10 client.?.objecter ms_handle_connect 0x7f86300a98f0 2024-03-25T17:58:19.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.461+0000 7f86177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:19.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.476+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3444692659 0 0) 0x7f863013be70 con 0x7f863013f710 2024-03-25T17:58:19.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.476+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86301317f0 con 0x7f863013f710 2024-03-25T17:58:19.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.476+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8624001930 con 0x7f863013f710 2024-03-25T17:58:19.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.476+0000 7f86177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:19.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.476+0000 7f86177fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:58:19.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.476+0000 7f86177fe640 10 monclient(hunting): dump: 2024-03-25T17:58:19.476 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:19.476 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:19.476 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:19.476 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:19.476 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:19.477 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:19.477 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:19.477 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:19.477 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:19.477 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:19.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.476+0000 7f86177fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:58:19.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.476+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345242354 0 0) 0x7f8630131660 con 0x7f86300a98f0 2024-03-25T17:58:19.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.476+0000 7f86177fe640 10 monclient(hunting): _negotiate 2024-03-25T17:58:19.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.476+0000 7f86177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:19.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.476+0000 7f86177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:19.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.476+0000 7f86177fe640 10 monclient(hunting): my global_id is 4303 2024-03-25T17:58:19.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.476+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f863013be70 con 0x7f86300a98f0 2024-03-25T17:58:19.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.488+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1357731402 0 0) 0x7f861c004eb0 con 0x7f86301430c0 2024-03-25T17:58:19.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.488+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8630131660 con 0x7f86301430c0 2024-03-25T17:58:19.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.488+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8628001b90 con 0x7f86301430c0 2024-03-25T17:58:19.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.488+0000 7f86177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:19.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.488+0000 7f86177fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:58:19.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.488+0000 7f86177fe640 10 monclient(hunting): dump: 2024-03-25T17:58:19.488 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:19.488 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:19.488 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:19.488 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:19.488 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:19.488 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:19.488 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:19.488 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:19.489 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:19.489 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:19.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1514852179 0 0) 0x7f86301317f0 con 0x7f863013f710 2024-03-25T17:58:19.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 >> v1:172.21.15.169:6789/0 conn(0x7f86300a98f0 legacy=0x7f8630130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:19.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 >> v1:172.21.15.18:6789/0 conn(0x7f86301430c0 legacy=0x7f86301e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:19.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f86177fe640 10 monclient: _finish_hunting 0 2024-03-25T17:58:19.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f86177fe640 1 monclient: found mon.c 2024-03-25T17:58:19.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f86177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:58:19.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f86177fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:58:19.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8630131980 con 0x7f863013f710 2024-03-25T17:58:19.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f86177fe640 10 monclient: _finish_auth 0 2024-03-25T17:58:19.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f86177fe640 10 monclient: _check_auth_tickets 2024-03-25T17:58:19.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f86177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:58:19.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f863530e640 5 monclient: authenticate success, global_id 4319 2024-03-25T17:58:19.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f863530e640 10 monclient: _renew_subs 2024-03-25T17:58:19.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f863530e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:58:19.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f863530e640 1 -- 172.21.15.18:0/2289722250 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8630135c60 con 0x7f863013f710 2024-03-25T17:58:19.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f863530e640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:58:19.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f863530e640 10 monclient: _renew_subs 2024-03-25T17:58:19.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f863530e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:58:19.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f863530e640 1 -- 172.21.15.18:0/2289722250 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86300b7770 con 0x7f863013f710 2024-03-25T17:58:19.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.489+0000 7f863530e640 1 librados: init done 2024-03-25T17:58:19.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.490+0000 7f863530e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:19.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.490+0000 7f863530e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:19.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.490+0000 7f863530e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:58:19.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.490+0000 7f863530e640 1 -- 172.21.15.18:0/2289722250 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86301371d0 con 0x7f863013f710 2024-03-25T17:58:19.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.811+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8624003610 con 0x7f863013f710 2024-03-25T17:58:19.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.811+0000 7f86177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:58:19.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.811+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8624003a20 con 0x7f863013f710 2024-03-25T17:58:19.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.811+0000 7f86177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:19.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.811+0000 7f86177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:58:19.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.811+0000 7f86177fe640 10 monclient: dump: 2024-03-25T17:58:19.811 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:19.811 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:19.811 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:19.811 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:19.812 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:19.812 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:19.812 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:19.812 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:19.812 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:19.812 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:19.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.811+0000 7f862d7fa640 4 set_mon_vals no callback set 2024-03-25T17:58:19.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.812+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f8624003c40 con 0x7f863013f710 2024-03-25T17:58:19.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.812+0000 7f862effd640 10 monclient: get_auth_request con 0x7f861c06e040 auth_method 0 2024-03-25T17:58:19.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.813+0000 7f86177fe640 10 client.4319.objecter ms_handle_connect 0x7f861c06e040 2024-03-25T17:58:19.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.819+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f8624053580 con 0x7f863013f710 2024-03-25T17:58:19.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.819+0000 7f86177fe640 10 client.4319.objecter ms_dispatch 0x7f8630000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:58:19.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.819+0000 7f86177fe640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:58:19.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.819+0000 7f86177fe640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:58:19.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.819+0000 7f86177fe640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-25T17:58:19.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.819+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f862404f090 con 0x7f863013f710 2024-03-25T17:58:19.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.819+0000 7f86177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:19.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:19.819+0000 7f86177fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:58:20.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.087+0000 7f863530e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:20.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.087+0000 7f863530e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:20.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.087+0000 7f863530e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:58:20.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.087+0000 7f863530e640 1 -- 172.21.15.18:0/2289722250 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8630074cc0 con 0x7f863013f710 2024-03-25T17:58:20.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.211+0000 7f86177fe640 1 -- 172.21.15.18:0/2289722250 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f8624050fc0 con 0x7f863013f710 2024-03-25T17:58:20.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.211+0000 7f86177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:20.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.211+0000 7f86177fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:58:20.211 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:58:20.211 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:58:20.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.213+0000 7f863530e640 10 librados: watch_flush enter 2024-03-25T17:58:20.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.213+0000 7f863530e640 10 librados: watch_flush exit 2024-03-25T17:58:20.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.214+0000 7f863530e640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-25T17:58:20.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.214+0000 7f863530e640 10 client.4319.objecter successfully canceled tick 2024-03-25T17:58:20.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.214+0000 7f863530e640 1 -- 172.21.15.18:0/2289722250 >> v1:172.21.15.18:6800/3721244451 conn(0x7f861c06e040 legacy=0x7f861c070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:20.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.214+0000 7f863530e640 10 monclient: shutdown 2024-03-25T17:58:20.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.214+0000 7f863530e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:58:20.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.214+0000 7f863530e640 1 -- 172.21.15.18:0/2289722250 >> v1:172.21.15.18:6790/0 conn(0x7f863013f710 legacy=0x7f86301d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:20.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.214+0000 7f863530e640 1 -- 172.21.15.18:0/2289722250 shutdown_connections 2024-03-25T17:58:20.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.214+0000 7f863530e640 1 -- 172.21.15.18:0/2289722250 >> 172.21.15.18:0/2289722250 conn(0x7f86300b7ea0 msgr2=0x7f8630074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:58:20.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.214+0000 7f863530e640 1 -- 172.21.15.18:0/2289722250 shutdown_connections 2024-03-25T17:58:20.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.214+0000 7f863530e640 1 -- 172.21.15.18:0/2289722250 wait complete. 2024-03-25T17:58:20.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:20.214+0000 7f863530e640 1 librados: shutdown 2024-03-25T17:58:20.226 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:58:22.181 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:02:19.974 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:22.237 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:02:20.030 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:22.271 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:02:20.065 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:22.309 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:02:20.102 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:22.311 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:02:20.061 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:22.319 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:02:20.112 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:22.320 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:02:20.069 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:22.346 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:02:20.095 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:22.349 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:02:20.142 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:27.033 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:58:27.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.148+0000 7f68868c0640 10 monclient: get_monmap_and_config 2024-03-25T17:58:27.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.148+0000 7f68868c0640 10 monclient: build_initial_monmap 2024-03-25T17:58:27.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.148+0000 7f68868c0640 10 monclient: monmap: 2024-03-25T17:58:27.152 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:58:27.152 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:27.152 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:58:27.148611+0000 2024-03-25T17:58:27.152 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:58:27.148611+0000 2024-03-25T17:58:27.152 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:58:27.152 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:27.152 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:58:27.152 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:58:27.152 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:58:27.152 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:27.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.148+0000 7f68868c0640 1 Processor -- start 2024-03-25T17:58:27.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.149+0000 7f68868c0640 1 -- start start 2024-03-25T17:58:27.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.149+0000 7f68868c0640 10 monclient: init 2024-03-25T17:58:27.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.149+0000 7f68868c0640 10 monclient: _reopen_session rank -1 2024-03-25T17:58:27.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.149+0000 7f68868c0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:58:27.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.149+0000 7f68868c0640 10 monclient(hunting): picked mon.noname-b con 0x7f68800b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T17:58:27.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.149+0000 7f68868c0640 10 monclient(hunting): picked mon.noname-a con 0x7f68800b1730 addr v1:172.21.15.18:6789/0 2024-03-25T17:58:27.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.149+0000 7f68868c0640 10 monclient(hunting): picked mon.noname-c con 0x7f688014be90 addr v1:172.21.15.169:6789/0 2024-03-25T17:58:27.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.149+0000 7f68868c0640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68800af940 con 0x7f68800b1730 2024-03-25T17:58:27.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.149+0000 7f68868c0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68800b0a70 con 0x7f688014be90 2024-03-25T17:58:27.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.149+0000 7f68868c0640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68800b0d50 con 0x7f68800b4d60 2024-03-25T17:58:27.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.149+0000 7f68868c0640 10 monclient(hunting): _renew_subs 2024-03-25T17:58:27.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.149+0000 7f68868c0640 10 monclient(hunting): authenticate will time out at 7035.727261s 2024-03-25T17:58:27.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.149+0000 7f687ffff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f68800b4d60 0x7f688013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:47934/0 (socket says 172.21.15.18:47934) 2024-03-25T17:58:27.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.149+0000 7f687ffff640 1 -- 172.21.15.18:0/646814214 learned_addr learned my addr 172.21.15.18:0/646814214 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:58:27.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.223+0000 7f687effd640 1 -- 172.21.15.18:0/646814214 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110868327 0 0) 0x7f68800af940 con 0x7f68800b1730 2024-03-25T17:58:27.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.223+0000 7f687effd640 10 monclient(hunting): _negotiate 2024-03-25T17:58:27.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.223+0000 7f687effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:27.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.223+0000 7f687effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:27.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.223+0000 7f687effd640 10 monclient(hunting): my global_id is 4314 2024-03-25T17:58:27.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.224+0000 7f687effd640 1 -- 172.21.15.18:0/646814214 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6868003520 con 0x7f68800b1730 2024-03-25T17:58:27.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.241+0000 7f687effd640 1 -- 172.21.15.18:0/646814214 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1422906616 0 0) 0x7f6868003520 con 0x7f68800b1730 2024-03-25T17:58:27.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.241+0000 7f687effd640 1 -- 172.21.15.18:0/646814214 >> v1:172.21.15.18:6790/0 conn(0x7f68800b4d60 legacy=0x7f688013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:27.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.241+0000 7f687effd640 1 -- 172.21.15.18:0/646814214 >> v1:172.21.15.169:6789/0 conn(0x7f688014be90 legacy=0x7f688014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:27.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.241+0000 7f687effd640 10 monclient: _finish_hunting 0 2024-03-25T17:58:27.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.241+0000 7f687effd640 1 monclient: found mon.noname-a 2024-03-25T17:58:27.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.241+0000 7f687effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:58:27.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.241+0000 7f687effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T17:58:27.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.241+0000 7f687effd640 1 -- 172.21.15.18:0/646814214 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68800b1030 con 0x7f68800b1730 2024-03-25T17:58:27.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.241+0000 7f687effd640 10 monclient: _finish_auth 0 2024-03-25T17:58:27.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.241+0000 7f687effd640 10 monclient: _check_auth_tickets 2024-03-25T17:58:27.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.241+0000 7f687effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:58:27.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.241+0000 7f687effd640 1 -- 172.21.15.18:0/646814214 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f6874001690 con 0x7f68800b1730 2024-03-25T17:58:27.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.241+0000 7f687effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:27.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.241+0000 7f687effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:58:27.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.241+0000 7f687effd640 10 monclient: dump: 2024-03-25T17:58:27.242 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:27.242 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:27.242 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:27.243 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:27.243 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:27.243 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:27.243 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:27.243 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:27.243 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:27.243 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:27.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.241+0000 7f68868c0640 5 monclient: authenticate success, global_id 4314 2024-03-25T17:58:27.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.241+0000 7f68868c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:58:27.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.256+0000 7f687effd640 1 -- 172.21.15.18:0/646814214 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68740017f0 con 0x7f68800b1730 2024-03-25T17:58:27.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.256+0000 7f687effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:58:27.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.256+0000 7f687effd640 1 -- 172.21.15.18:0/646814214 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f6874001c00 con 0x7f68800b1730 2024-03-25T17:58:27.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.256+0000 7f687effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:27.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.256+0000 7f687effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:58:27.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.256+0000 7f687effd640 10 monclient: dump: 2024-03-25T17:58:27.257 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:27.257 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:27.257 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:27.257 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:27.257 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:27.257 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:27.257 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:27.257 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:27.258 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:27.258 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:27.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.256+0000 7f68868c0640 10 monclient: get_monmap_and_config success 2024-03-25T17:58:27.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.256+0000 7f68868c0640 4 set_mon_vals no callback set 2024-03-25T17:58:27.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.256+0000 7f68868c0640 10 monclient: shutdown 2024-03-25T17:58:27.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.256+0000 7f68868c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:58:27.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.256+0000 7f68868c0640 1 -- 172.21.15.18:0/646814214 >> v1:172.21.15.18:6789/0 conn(0x7f68800b1730 legacy=0x7f68800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:27.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.257+0000 7f68868c0640 1 -- 172.21.15.18:0/646814214 shutdown_connections 2024-03-25T17:58:27.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.257+0000 7f68868c0640 1 -- 172.21.15.18:0/646814214 >> 172.21.15.18:0/646814214 conn(0x7f68800617a0 msgr2=0x7f6880061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:58:27.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.257+0000 7f68868c0640 1 -- 172.21.15.18:0/646814214 shutdown_connections 2024-03-25T17:58:27.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.257+0000 7f68868c0640 1 -- 172.21.15.18:0/646814214 wait complete. 2024-03-25T17:58:27.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.257+0000 7f68868c0640 10 monclient: build_initial_monmap 2024-03-25T17:58:27.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.257+0000 7f68868c0640 10 monclient: monmap: 2024-03-25T17:58:27.260 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:58:27.260 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:58:27.260 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:58:27.260 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:58:27.260 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:58:27.260 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:58:27.260 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:58:27.260 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:58:27.260 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:58:27.261 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:27.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.257+0000 7f68868c0640 1 librados: starting msgr at 2024-03-25T17:58:27.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.257+0000 7f68868c0640 1 librados: starting objecter 2024-03-25T17:58:27.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.258+0000 7f68868c0640 1 Processor -- start 2024-03-25T17:58:27.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.258+0000 7f68868c0640 1 -- start start 2024-03-25T17:58:27.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.258+0000 7f68868c0640 1 librados: setting wanted keys 2024-03-25T17:58:27.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.258+0000 7f68868c0640 1 librados: calling monclient init 2024-03-25T17:58:27.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.258+0000 7f68868c0640 10 monclient: init 2024-03-25T17:58:27.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.258+0000 7f68868c0640 10 monclient: _reopen_session rank -1 2024-03-25T17:58:27.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.258+0000 7f68868c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:58:27.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.258+0000 7f68868c0640 10 monclient(hunting): picked mon.noname-c con 0x7f68800b1730 addr v1:172.21.15.18:6790/0 2024-03-25T17:58:27.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.258+0000 7f68868c0640 10 monclient(hunting): picked mon.noname-b con 0x7f68800b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T17:58:27.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.258+0000 7f68868c0640 10 monclient(hunting): picked mon.noname-a con 0x7f688014be90 addr v1:172.21.15.18:6789/0 2024-03-25T17:58:27.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.258+0000 7f68868c0640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6880140380 con 0x7f688014be90 2024-03-25T17:58:27.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.258+0000 7f68868c0640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6880141ca0 con 0x7f68800b4d60 2024-03-25T17:58:27.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.258+0000 7f68868c0640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6880140860 con 0x7f68800b1730 2024-03-25T17:58:27.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.258+0000 7f68868c0640 10 monclient(hunting): _renew_subs 2024-03-25T17:58:27.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.258+0000 7f68868c0640 10 monclient(hunting): authenticate will time out at 7035.836616s 2024-03-25T17:58:27.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.261+0000 7f687f7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f68800b4d60 0x7f68801db720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:33926/0 (socket says 172.21.15.18:33926) 2024-03-25T17:58:27.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.261+0000 7f687f7fe640 1 -- 172.21.15.18:0/1700314341 learned_addr learned my addr 172.21.15.18:0/1700314341 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:58:27.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.262+0000 7f687cff9640 10 client.?.objecter ms_handle_connect 0x7f68800b4d60 2024-03-25T17:58:27.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.262+0000 7f687cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:27.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.275+0000 7f687cff9640 10 client.?.objecter ms_handle_connect 0x7f68800b1730 2024-03-25T17:58:27.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.275+0000 7f687cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:27.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.281+0000 7f687cff9640 10 client.?.objecter ms_handle_connect 0x7f688014be90 2024-03-25T17:58:27.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.281+0000 7f687cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:27.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.345+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411129978 0 0) 0x7f6880140380 con 0x7f688014be90 2024-03-25T17:58:27.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.345+0000 7f687cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:58:27.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.345+0000 7f687cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:27.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.345+0000 7f687cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:27.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.345+0000 7f687cff9640 10 monclient(hunting): my global_id is 4317 2024-03-25T17:58:27.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.345+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f686c004eb0 con 0x7f688014be90 2024-03-25T17:58:27.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.354+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191275537 0 0) 0x7f6880140860 con 0x7f68800b1730 2024-03-25T17:58:27.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.354+0000 7f687cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:58:27.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.354+0000 7f687cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:27.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.354+0000 7f687cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:27.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.354+0000 7f687cff9640 10 monclient(hunting): my global_id is 4325 2024-03-25T17:58:27.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.354+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6880140380 con 0x7f68800b1730 2024-03-25T17:58:27.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.369+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 675939690 0 0) 0x7f686c004eb0 con 0x7f688014be90 2024-03-25T17:58:27.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.369+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6880140860 con 0x7f688014be90 2024-03-25T17:58:27.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.369+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f6878001b20 con 0x7f688014be90 2024-03-25T17:58:27.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.369+0000 7f687cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:27.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.369+0000 7f687cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:58:27.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.369+0000 7f687cff9640 10 monclient(hunting): dump: 2024-03-25T17:58:27.370 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:27.370 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:27.370 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:27.370 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:27.370 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:27.370 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:27.370 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:27.370 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:27.370 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:27.370 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:27.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.369+0000 7f687cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:58:27.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.369+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1654899379 0 0) 0x7f6880140380 con 0x7f68800b1730 2024-03-25T17:58:27.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.369+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f686c004eb0 con 0x7f68800b1730 2024-03-25T17:58:27.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.370+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f68700018a0 con 0x7f68800b1730 2024-03-25T17:58:27.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.370+0000 7f687cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:27.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.370+0000 7f687cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:58:27.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.370+0000 7f687cff9640 10 monclient(hunting): dump: 2024-03-25T17:58:27.371 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:27.371 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:27.371 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:27.371 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:27.371 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:27.371 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:27.371 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:27.371 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:27.371 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:27.372 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:27.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3514952862 0 0) 0x7f6880140860 con 0x7f688014be90 2024-03-25T17:58:27.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 >> v1:172.21.15.18:6790/0 conn(0x7f68800b1730 legacy=0x7f68801db010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:27.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 >> v1:172.21.15.169:6789/0 conn(0x7f68800b4d60 legacy=0x7f68801db720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:27.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f687cff9640 10 monclient: _finish_hunting 0 2024-03-25T17:58:27.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f687cff9640 1 monclient: found mon.a 2024-03-25T17:58:27.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f687cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:58:27.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f687cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:58:27.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68801423a0 con 0x7f688014be90 2024-03-25T17:58:27.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f687cff9640 10 monclient: _finish_auth 0 2024-03-25T17:58:27.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f687cff9640 10 monclient: _check_auth_tickets 2024-03-25T17:58:27.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f687cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:58:27.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f68868c0640 5 monclient: authenticate success, global_id 4317 2024-03-25T17:58:27.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f68868c0640 10 monclient: _renew_subs 2024-03-25T17:58:27.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f68868c0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:58:27.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f68868c0640 1 -- 172.21.15.18:0/1700314341 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6880138650 con 0x7f688014be90 2024-03-25T17:58:27.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f68868c0640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:58:27.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f68868c0640 10 monclient: _renew_subs 2024-03-25T17:58:27.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f68868c0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:58:27.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f68868c0640 1 -- 172.21.15.18:0/1700314341 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6880140d00 con 0x7f688014be90 2024-03-25T17:58:27.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.386+0000 7f68868c0640 1 librados: init done 2024-03-25T17:58:27.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.387+0000 7f68868c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:27.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.387+0000 7f68868c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:27.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.387+0000 7f68868c0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:58:27.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.387+0000 7f68868c0640 1 -- 172.21.15.18:0/1700314341 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68800627b0 con 0x7f688014be90 2024-03-25T17:58:27.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.703+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6878002030 con 0x7f688014be90 2024-03-25T17:58:27.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.703+0000 7f687cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:58:27.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.703+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f6878003b10 con 0x7f688014be90 2024-03-25T17:58:27.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.703+0000 7f687cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:27.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.703+0000 7f687cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:58:27.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.703+0000 7f687cff9640 10 monclient: dump: 2024-03-25T17:58:27.703 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:27.703 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:27.704 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:27.704 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:27.704 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:27.704 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:27.704 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:27.704 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:27.704 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:27.704 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:27.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.703+0000 7f687e7fc640 4 set_mon_vals no callback set 2024-03-25T17:58:27.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.704+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f6878019ae0 con 0x7f688014be90 2024-03-25T17:58:27.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.704+0000 7f687ffff640 10 monclient: get_auth_request con 0x7f686c06df20 auth_method 0 2024-03-25T17:58:27.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.705+0000 7f687cff9640 10 client.4317.objecter ms_handle_connect 0x7f686c06df20 2024-03-25T17:58:27.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.712+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f68780536b0 con 0x7f688014be90 2024-03-25T17:58:27.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.712+0000 7f687cff9640 10 client.4317.objecter ms_dispatch 0x7f6880000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:58:27.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.712+0000 7f687cff9640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:58:27.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.712+0000 7f687cff9640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:58:27.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.712+0000 7f687cff9640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-25T17:58:27.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.712+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f687804f1c0 con 0x7f688014be90 2024-03-25T17:58:27.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.712+0000 7f687cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:27.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.712+0000 7f687cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:58:27.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.961+0000 7f68868c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:27.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.961+0000 7f68868c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:27.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.961+0000 7f68868c0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:58:27.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.961+0000 7f68868c0640 1 -- 172.21.15.18:0/1700314341 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6880069f40 con 0x7f688014be90 2024-03-25T17:58:27.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.986+0000 7f687cff9640 1 -- 172.21.15.18:0/1700314341 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f68780510f0 con 0x7f688014be90 2024-03-25T17:58:27.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.986+0000 7f687cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:27.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.986+0000 7f687cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:58:27.987 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:58:27.987 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:58:27.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.988+0000 7f68868c0640 10 librados: watch_flush enter 2024-03-25T17:58:27.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.988+0000 7f68868c0640 10 librados: watch_flush exit 2024-03-25T17:58:27.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.988+0000 7f68868c0640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-25T17:58:27.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.988+0000 7f68868c0640 10 client.4317.objecter successfully canceled tick 2024-03-25T17:58:27.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.989+0000 7f68868c0640 1 -- 172.21.15.18:0/1700314341 >> v1:172.21.15.18:6800/3721244451 conn(0x7f686c06df20 legacy=0x7f686c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:27.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.989+0000 7f68868c0640 10 monclient: shutdown 2024-03-25T17:58:27.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.989+0000 7f68868c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:58:27.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.989+0000 7f68868c0640 1 -- 172.21.15.18:0/1700314341 >> v1:172.21.15.18:6789/0 conn(0x7f688014be90 legacy=0x7f68801e22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:27.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.989+0000 7f68868c0640 1 -- 172.21.15.18:0/1700314341 shutdown_connections 2024-03-25T17:58:27.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.989+0000 7f68868c0640 1 -- 172.21.15.18:0/1700314341 >> 172.21.15.18:0/1700314341 conn(0x7f68800617a0 msgr2=0x7f688013c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:58:27.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.989+0000 7f68868c0640 1 -- 172.21.15.18:0/1700314341 shutdown_connections 2024-03-25T17:58:27.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.989+0000 7f68868c0640 1 -- 172.21.15.18:0/1700314341 wait complete. 2024-03-25T17:58:27.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:27.989+0000 7f68868c0640 1 librados: shutdown 2024-03-25T17:58:28.002 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:58:34.810 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:58:34.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.931+0000 7fce7493d640 10 monclient: get_monmap_and_config 2024-03-25T17:58:34.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.931+0000 7fce7493d640 10 monclient: build_initial_monmap 2024-03-25T17:58:34.932 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.931+0000 7fce7493d640 10 monclient: monmap: 2024-03-25T17:58:34.932 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:58:34.932 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:34.932 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:58:34.932205+0000 2024-03-25T17:58:34.932 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:58:34.932205+0000 2024-03-25T17:58:34.932 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:58:34.932 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:34.932 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:58:34.932 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:58:34.932 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:58:34.932 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:34.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.932+0000 7fce7493d640 1 Processor -- start 2024-03-25T17:58:34.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.932+0000 7fce7493d640 1 -- start start 2024-03-25T17:58:34.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.932+0000 7fce7493d640 10 monclient: init 2024-03-25T17:58:34.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.933+0000 7fce7493d640 10 monclient: _reopen_session rank -1 2024-03-25T17:58:34.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.933+0000 7fce7493d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:58:34.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.933+0000 7fce7493d640 10 monclient(hunting): picked mon.noname-a con 0x7fce7006ea40 addr v1:172.21.15.18:6789/0 2024-03-25T17:58:34.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.933+0000 7fce7493d640 10 monclient(hunting): picked mon.noname-c con 0x7fce70071800 addr v1:172.21.15.169:6789/0 2024-03-25T17:58:34.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.933+0000 7fce7493d640 10 monclient(hunting): picked mon.noname-b con 0x7fce70071c40 addr v1:172.21.15.18:6790/0 2024-03-25T17:58:34.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.933+0000 7fce7493d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce7006f1f0 con 0x7fce7006ea40 2024-03-25T17:58:34.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.933+0000 7fce7493d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce70070d00 con 0x7fce70071800 2024-03-25T17:58:34.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.933+0000 7fce7493d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce700720b0 con 0x7fce70071c40 2024-03-25T17:58:34.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.933+0000 7fce7493d640 10 monclient(hunting): _renew_subs 2024-03-25T17:58:34.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.933+0000 7fce7493d640 10 monclient(hunting): authenticate will time out at 7043.511482s 2024-03-25T17:58:34.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.943+0000 7fce67fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fce70071800 0x7fce70147070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:54084/0 (socket says 172.21.15.18:54084) 2024-03-25T17:58:34.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.943+0000 7fce67fff640 1 -- 172.21.15.18:0/1576888160 learned_addr learned my addr 172.21.15.18:0/1576888160 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:58:34.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.953+0000 7fce6dd9b640 1 -- 172.21.15.18:0/1576888160 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183678264 0 0) 0x7fce70070d00 con 0x7fce70071800 2024-03-25T17:58:34.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.953+0000 7fce6dd9b640 10 monclient(hunting): _negotiate 2024-03-25T17:58:34.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.953+0000 7fce6dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:34.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.953+0000 7fce6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:34.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.953+0000 7fce6dd9b640 10 monclient(hunting): my global_id is 4312 2024-03-25T17:58:34.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.953+0000 7fce6dd9b640 1 -- 172.21.15.18:0/1576888160 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce5c003520 con 0x7fce70071800 2024-03-25T17:58:34.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.965+0000 7fce6dd9b640 1 -- 172.21.15.18:0/1576888160 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053537317 0 0) 0x7fce7006f1f0 con 0x7fce7006ea40 2024-03-25T17:58:34.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.965+0000 7fce6dd9b640 10 monclient(hunting): _negotiate 2024-03-25T17:58:34.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.965+0000 7fce6dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:34.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.965+0000 7fce6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:34.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.965+0000 7fce6dd9b640 10 monclient(hunting): my global_id is 4320 2024-03-25T17:58:34.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.965+0000 7fce6dd9b640 1 -- 172.21.15.18:0/1576888160 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce70070d00 con 0x7fce7006ea40 2024-03-25T17:58:34.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.969+0000 7fce6dd9b640 1 -- 172.21.15.18:0/1576888160 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583279557 0 0) 0x7fce700720b0 con 0x7fce70071c40 2024-03-25T17:58:34.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.969+0000 7fce6dd9b640 10 monclient(hunting): _negotiate 2024-03-25T17:58:34.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.969+0000 7fce6dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:34.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.969+0000 7fce6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:34.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.969+0000 7fce6dd9b640 10 monclient(hunting): my global_id is 4328 2024-03-25T17:58:34.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.969+0000 7fce6dd9b640 1 -- 172.21.15.18:0/1576888160 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce7006f1f0 con 0x7fce70071c40 2024-03-25T17:58:34.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.979+0000 7fce6dd9b640 1 -- 172.21.15.18:0/1576888160 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3057629700 0 0) 0x7fce5c003520 con 0x7fce70071800 2024-03-25T17:58:34.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.979+0000 7fce6dd9b640 1 -- 172.21.15.18:0/1576888160 >> v1:172.21.15.18:6790/0 conn(0x7fce70071c40 legacy=0x7fce7014a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:34.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.979+0000 7fce6dd9b640 1 -- 172.21.15.18:0/1576888160 >> v1:172.21.15.18:6789/0 conn(0x7fce7006ea40 legacy=0x7fce7013ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:34.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.979+0000 7fce6dd9b640 10 monclient: _finish_hunting 0 2024-03-25T17:58:34.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.979+0000 7fce6dd9b640 1 monclient: found mon.noname-c 2024-03-25T17:58:34.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.979+0000 7fce6dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:58:34.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.979+0000 7fce6dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T17:58:34.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.979+0000 7fce6dd9b640 1 -- 172.21.15.18:0/1576888160 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce70072780 con 0x7fce70071800 2024-03-25T17:58:34.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.979+0000 7fce6dd9b640 10 monclient: _finish_auth 0 2024-03-25T17:58:34.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.979+0000 7fce6dd9b640 10 monclient: _check_auth_tickets 2024-03-25T17:58:34.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.979+0000 7fce6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:58:34.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.979+0000 7fce6dd9b640 1 -- 172.21.15.18:0/1576888160 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fce58001690 con 0x7fce70071800 2024-03-25T17:58:34.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.979+0000 7fce6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:34.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.979+0000 7fce6dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:58:34.980 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.979+0000 7fce6dd9b640 10 monclient: dump: 2024-03-25T17:58:34.981 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:34.981 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:34.981 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:34.981 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:34.981 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:34.981 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:34.981 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:34.981 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:34.981 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:34.981 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:34.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.979+0000 7fce7493d640 5 monclient: authenticate success, global_id 4312 2024-03-25T17:58:34.981 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.979+0000 7fce7493d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:58:34.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.987+0000 7fce6dd9b640 1 -- 172.21.15.18:0/1576888160 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce580017f0 con 0x7fce70071800 2024-03-25T17:58:34.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.987+0000 7fce6dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:58:34.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.987+0000 7fce6dd9b640 1 -- 172.21.15.18:0/1576888160 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fce58001c00 con 0x7fce70071800 2024-03-25T17:58:34.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.987+0000 7fce7493d640 10 monclient: get_monmap_and_config success 2024-03-25T17:58:34.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.987+0000 7fce7493d640 4 set_mon_vals no callback set 2024-03-25T17:58:34.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.987+0000 7fce6dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:34.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.987+0000 7fce6dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:58:34.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.987+0000 7fce6dd9b640 10 monclient: dump: 2024-03-25T17:58:34.989 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:34.989 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:34.989 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:34.989 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:34.989 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:34.989 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:34.989 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:34.989 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:34.989 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:34.989 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:34.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.988+0000 7fce7493d640 10 monclient: shutdown 2024-03-25T17:58:34.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.988+0000 7fce7493d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:58:34.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.988+0000 7fce7493d640 1 -- 172.21.15.18:0/1576888160 >> v1:172.21.15.169:6789/0 conn(0x7fce70071800 legacy=0x7fce70147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:34.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.988+0000 7fce7493d640 1 -- 172.21.15.18:0/1576888160 shutdown_connections 2024-03-25T17:58:34.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.988+0000 7fce7493d640 1 -- 172.21.15.18:0/1576888160 >> 172.21.15.18:0/1576888160 conn(0x7fce7006b630 msgr2=0x7fce700b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:58:34.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.988+0000 7fce7493d640 1 -- 172.21.15.18:0/1576888160 shutdown_connections 2024-03-25T17:58:34.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.988+0000 7fce7493d640 1 -- 172.21.15.18:0/1576888160 wait complete. 2024-03-25T17:58:34.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.989+0000 7fce7493d640 10 monclient: build_initial_monmap 2024-03-25T17:58:34.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.989+0000 7fce7493d640 10 monclient: monmap: 2024-03-25T17:58:34.992 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:58:34.992 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:58:34.992 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:58:34.992 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:58:34.992 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:58:34.992 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:58:34.992 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:58:34.992 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:58:34.993 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:58:34.993 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:34.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.990+0000 7fce7493d640 1 librados: starting msgr at 2024-03-25T17:58:34.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.990+0000 7fce7493d640 1 librados: starting objecter 2024-03-25T17:58:34.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.990+0000 7fce7493d640 1 Processor -- start 2024-03-25T17:58:34.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.990+0000 7fce7493d640 1 -- start start 2024-03-25T17:58:34.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.990+0000 7fce7493d640 1 librados: setting wanted keys 2024-03-25T17:58:34.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.990+0000 7fce7493d640 1 librados: calling monclient init 2024-03-25T17:58:34.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.990+0000 7fce7493d640 10 monclient: init 2024-03-25T17:58:34.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.990+0000 7fce7493d640 10 monclient: _reopen_session rank -1 2024-03-25T17:58:34.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.990+0000 7fce7493d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:58:34.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.990+0000 7fce7493d640 10 monclient(hunting): picked mon.noname-b con 0x7fce7006ea40 addr v1:172.21.15.169:6789/0 2024-03-25T17:58:34.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.990+0000 7fce7493d640 10 monclient(hunting): picked mon.noname-c con 0x7fce70071800 addr v1:172.21.15.18:6790/0 2024-03-25T17:58:34.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.991+0000 7fce7493d640 10 monclient(hunting): picked mon.noname-a con 0x7fce70071c40 addr v1:172.21.15.18:6789/0 2024-03-25T17:58:34.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.991+0000 7fce7493d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce700ac780 con 0x7fce70071c40 2024-03-25T17:58:34.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.991+0000 7fce7493d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce700ab290 con 0x7fce7006ea40 2024-03-25T17:58:34.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.991+0000 7fce7493d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce700ab500 con 0x7fce70071800 2024-03-25T17:58:34.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.991+0000 7fce7493d640 10 monclient(hunting): _renew_subs 2024-03-25T17:58:34.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.991+0000 7fce7493d640 10 monclient(hunting): authenticate will time out at 7043.568703s 2024-03-25T17:58:34.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.997+0000 7fce67fff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fce70071800 0x7fce701d8060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:35530/0 (socket says 172.21.15.18:35530) 2024-03-25T17:58:34.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.997+0000 7fce67fff640 1 -- 172.21.15.18:0/1912709073 learned_addr learned my addr 172.21.15.18:0/1912709073 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:58:34.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.999+0000 7fce66ffd640 10 client.?.objecter ms_handle_connect 0x7fce70071800 2024-03-25T17:58:34.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:34.999+0000 7fce66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:35.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.001+0000 7fce66ffd640 10 client.?.objecter ms_handle_connect 0x7fce70071c40 2024-03-25T17:58:35.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.001+0000 7fce66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:35.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.004+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435780381 0 0) 0x7fce700ab500 con 0x7fce70071800 2024-03-25T17:58:35.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.004+0000 7fce66ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:58:35.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.004+0000 7fce66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:35.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.004+0000 7fce66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:35.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.004+0000 7fce66ffd640 10 monclient(hunting): my global_id is 4331 2024-03-25T17:58:35.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.004+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce50004eb0 con 0x7fce70071800 2024-03-25T17:58:35.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.008+0000 7fce66ffd640 10 client.?.objecter ms_handle_connect 0x7fce7006ea40 2024-03-25T17:58:35.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.008+0000 7fce66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:35.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.016+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2863826425 0 0) 0x7fce50004eb0 con 0x7fce70071800 2024-03-25T17:58:35.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.016+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce700ab500 con 0x7fce70071800 2024-03-25T17:58:35.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.016+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fce58001bc0 con 0x7fce70071800 2024-03-25T17:58:35.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.016+0000 7fce66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:35.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.016+0000 7fce66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:58:35.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.016+0000 7fce66ffd640 10 monclient(hunting): dump: 2024-03-25T17:58:35.017 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:35.017 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:35.017 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:35.017 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:35.017 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:35.017 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:35.017 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:35.017 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:35.017 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:35.017 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:35.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.016+0000 7fce66ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:58:35.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.017+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915961707 0 0) 0x7fce700ac780 con 0x7fce70071c40 2024-03-25T17:58:35.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.017+0000 7fce66ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:58:35.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.017+0000 7fce66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:35.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.017+0000 7fce66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:35.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.017+0000 7fce66ffd640 10 monclient(hunting): my global_id is 4323 2024-03-25T17:58:35.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.017+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce50004eb0 con 0x7fce70071c40 2024-03-25T17:58:35.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.036+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3304363408 0 0) 0x7fce50004eb0 con 0x7fce70071c40 2024-03-25T17:58:35.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.036+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce700ac780 con 0x7fce70071c40 2024-03-25T17:58:35.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.036+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fce68001bd0 con 0x7fce70071c40 2024-03-25T17:58:35.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.036+0000 7fce66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:35.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.036+0000 7fce66ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:58:35.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.036+0000 7fce66ffd640 10 monclient(hunting): dump: 2024-03-25T17:58:35.037 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:35.037 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:35.037 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:35.037 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:35.037 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:35.037 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:35.037 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:35.037 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:35.037 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:35.037 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:35.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1173143922 0 0) 0x7fce700ac780 con 0x7fce70071c40 2024-03-25T17:58:35.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 >> v1:172.21.15.18:6790/0 conn(0x7fce70071800 legacy=0x7fce701d8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:35.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 >> v1:172.21.15.169:6789/0 conn(0x7fce7006ea40 legacy=0x7fce701d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:35.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce66ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:58:35.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce66ffd640 1 monclient: found mon.a 2024-03-25T17:58:35.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:58:35.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce66ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:58:35.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce70138d30 con 0x7fce70071c40 2024-03-25T17:58:35.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce66ffd640 10 monclient: _finish_auth 0 2024-03-25T17:58:35.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce66ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:58:35.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:58:35.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce7493d640 5 monclient: authenticate success, global_id 4323 2024-03-25T17:58:35.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce7493d640 10 monclient: _renew_subs 2024-03-25T17:58:35.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce7493d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:58:35.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce7493d640 1 -- 172.21.15.18:0/1912709073 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce700ac1e0 con 0x7fce70071c40 2024-03-25T17:58:35.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce7493d640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:58:35.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce7493d640 10 monclient: _renew_subs 2024-03-25T17:58:35.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce7493d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:58:35.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce7493d640 1 -- 172.21.15.18:0/1912709073 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce701c91b0 con 0x7fce70071c40 2024-03-25T17:58:35.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.052+0000 7fce7493d640 1 librados: init done 2024-03-25T17:58:35.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.053+0000 7fce7493d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:35.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.053+0000 7fce7493d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:35.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.053+0000 7fce7493d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:58:35.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.053+0000 7fce7493d640 1 -- 172.21.15.18:0/1912709073 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce700ac530 con 0x7fce70071c40 2024-03-25T17:58:35.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.370+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce68001fe0 con 0x7fce70071c40 2024-03-25T17:58:35.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.370+0000 7fce66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:58:35.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.370+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fce68003b30 con 0x7fce70071c40 2024-03-25T17:58:35.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.370+0000 7fce66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:35.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.370+0000 7fce66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:58:35.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.370+0000 7fce66ffd640 10 monclient: dump: 2024-03-25T17:58:35.370 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:35.371 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:35.371 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:35.371 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:35.371 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:35.371 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:35.371 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:35.371 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:35.371 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:35.371 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:35.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.370+0000 7fce6d59a640 4 set_mon_vals no callback set 2024-03-25T17:58:35.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.371+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fce68019b00 con 0x7fce70071c40 2024-03-25T17:58:35.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.371+0000 7fce6e59c640 10 monclient: get_auth_request con 0x7fce5006ddf0 auth_method 0 2024-03-25T17:58:35.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.372+0000 7fce66ffd640 10 client.4323.objecter ms_handle_connect 0x7fce5006ddf0 2024-03-25T17:58:35.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.378+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fce680537d0 con 0x7fce70071c40 2024-03-25T17:58:35.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.378+0000 7fce66ffd640 10 client.4323.objecter ms_dispatch 0x7fce70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:58:35.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.378+0000 7fce66ffd640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:58:35.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.378+0000 7fce66ffd640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:58:35.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.378+0000 7fce66ffd640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-25T17:58:35.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.378+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce6804f110 con 0x7fce70071c40 2024-03-25T17:58:35.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.378+0000 7fce66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:35.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.378+0000 7fce66ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:58:35.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.627+0000 7fce7493d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:35.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.627+0000 7fce7493d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:35.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.627+0000 7fce7493d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:58:35.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.627+0000 7fce7493d640 1 -- 172.21.15.18:0/1912709073 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce70074d80 con 0x7fce70071c40 2024-03-25T17:58:35.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.651+0000 7fce66ffd640 1 -- 172.21.15.18:0/1912709073 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fce70074d80 con 0x7fce70071c40 2024-03-25T17:58:35.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.651+0000 7fce66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:35.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.651+0000 7fce66ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:58:35.652 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:58:35.652 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:58:35.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.654+0000 7fce7493d640 10 librados: watch_flush enter 2024-03-25T17:58:35.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.654+0000 7fce7493d640 10 librados: watch_flush exit 2024-03-25T17:58:35.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.654+0000 7fce7493d640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-25T17:58:35.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.654+0000 7fce7493d640 10 client.4323.objecter successfully canceled tick 2024-03-25T17:58:35.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.654+0000 7fce7493d640 1 -- 172.21.15.18:0/1912709073 >> v1:172.21.15.18:6800/3721244451 conn(0x7fce5006ddf0 legacy=0x7fce500702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:35.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.654+0000 7fce7493d640 10 monclient: shutdown 2024-03-25T17:58:35.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.654+0000 7fce7493d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:58:35.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.654+0000 7fce7493d640 1 -- 172.21.15.18:0/1912709073 >> v1:172.21.15.18:6789/0 conn(0x7fce70071c40 legacy=0x7fce701e2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:35.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.654+0000 7fce7493d640 1 -- 172.21.15.18:0/1912709073 shutdown_connections 2024-03-25T17:58:35.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.654+0000 7fce7493d640 1 -- 172.21.15.18:0/1912709073 >> 172.21.15.18:0/1912709073 conn(0x7fce7006b630 msgr2=0x7fce7013dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:58:35.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.655+0000 7fce7493d640 1 -- 172.21.15.18:0/1912709073 shutdown_connections 2024-03-25T17:58:35.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.655+0000 7fce7493d640 1 -- 172.21.15.18:0/1912709073 wait complete. 2024-03-25T17:58:35.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:35.655+0000 7fce7493d640 1 librados: shutdown 2024-03-25T17:58:35.666 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:58:42.362 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:02:40.155 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:42.369 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:02:40.162 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:42.407 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:02:40.200 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:42.425 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:02:40.174 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:42.428 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:02:40.178 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:42.440 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:02:40.233 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:42.444 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:02:40.237 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:42.456 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:02:40.206 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:42.473 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:58:42.484 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:02:40.277 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:58:42.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.588+0000 7fa8477cf640 10 monclient: get_monmap_and_config 2024-03-25T17:58:42.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.588+0000 7fa8477cf640 10 monclient: build_initial_monmap 2024-03-25T17:58:42.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.588+0000 7fa8477cf640 10 monclient: monmap: 2024-03-25T17:58:42.589 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:58:42.589 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:42.589 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:58:42.589893+0000 2024-03-25T17:58:42.589 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:58:42.589893+0000 2024-03-25T17:58:42.590 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:58:42.590 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:42.590 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:58:42.590 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:58:42.590 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:58:42.590 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:42.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.589+0000 7fa8477cf640 1 Processor -- start 2024-03-25T17:58:42.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.589+0000 7fa8477cf640 1 -- start start 2024-03-25T17:58:42.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.589+0000 7fa8477cf640 10 monclient: init 2024-03-25T17:58:42.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.590+0000 7fa8477cf640 10 monclient: _reopen_session rank -1 2024-03-25T17:58:42.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.590+0000 7fa8477cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:58:42.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.590+0000 7fa8477cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa84006c380 addr v1:172.21.15.18:6790/0 2024-03-25T17:58:42.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.590+0000 7fa8477cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa8400b12d0 addr v1:172.21.15.169:6789/0 2024-03-25T17:58:42.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.590+0000 7fa8477cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa8400aedf0 addr v1:172.21.15.18:6789/0 2024-03-25T17:58:42.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.590+0000 7fa8477cf640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8400b16f0 con 0x7fa8400aedf0 2024-03-25T17:58:42.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.590+0000 7fa8477cf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8400adfe0 con 0x7fa8400b12d0 2024-03-25T17:58:42.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.590+0000 7fa8477cf640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8400add00 con 0x7fa84006c380 2024-03-25T17:58:42.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.590+0000 7fa8477cf640 10 monclient(hunting): _renew_subs 2024-03-25T17:58:42.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.590+0000 7fa8477cf640 10 monclient(hunting): authenticate will time out at 7051.169059s 2024-03-25T17:58:42.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.607+0000 7fa844d6a640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fa8400b12d0 0x7fa8400ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:58488/0 (socket says 172.21.15.18:58488) 2024-03-25T17:58:42.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.607+0000 7fa844d6a640 1 -- 172.21.15.18:0/3539397647 learned_addr learned my addr 172.21.15.18:0/3539397647 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:58:42.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.627+0000 7fa837fff640 1 -- 172.21.15.18:0/3539397647 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336220408 0 0) 0x7fa8400adfe0 con 0x7fa8400b12d0 2024-03-25T17:58:42.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.627+0000 7fa837fff640 10 monclient(hunting): _negotiate 2024-03-25T17:58:42.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.627+0000 7fa837fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:42.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.627+0000 7fa837fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:42.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.627+0000 7fa837fff640 10 monclient(hunting): my global_id is 4315 2024-03-25T17:58:42.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.627+0000 7fa837fff640 1 -- 172.21.15.18:0/3539397647 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa82c003520 con 0x7fa8400b12d0 2024-03-25T17:58:42.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.633+0000 7fa837fff640 1 -- 172.21.15.18:0/3539397647 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563014019 0 0) 0x7fa8400add00 con 0x7fa84006c380 2024-03-25T17:58:42.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.633+0000 7fa837fff640 10 monclient(hunting): _negotiate 2024-03-25T17:58:42.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.633+0000 7fa837fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:42.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.633+0000 7fa837fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:42.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.633+0000 7fa837fff640 10 monclient(hunting): my global_id is 4334 2024-03-25T17:58:42.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.633+0000 7fa837fff640 1 -- 172.21.15.18:0/3539397647 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8400adfe0 con 0x7fa84006c380 2024-03-25T17:58:42.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.636+0000 7fa837fff640 1 -- 172.21.15.18:0/3539397647 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327741643 0 0) 0x7fa8400b16f0 con 0x7fa8400aedf0 2024-03-25T17:58:42.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.636+0000 7fa837fff640 10 monclient(hunting): _negotiate 2024-03-25T17:58:42.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.636+0000 7fa837fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:42.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.636+0000 7fa837fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:42.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.636+0000 7fa837fff640 10 monclient(hunting): my global_id is 4326 2024-03-25T17:58:42.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.636+0000 7fa837fff640 1 -- 172.21.15.18:0/3539397647 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8400add00 con 0x7fa8400aedf0 2024-03-25T17:58:42.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa837fff640 1 -- 172.21.15.18:0/3539397647 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 528778150 0 0) 0x7fa82c003520 con 0x7fa8400b12d0 2024-03-25T17:58:42.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa837fff640 1 -- 172.21.15.18:0/3539397647 >> v1:172.21.15.18:6790/0 conn(0x7fa84006c380 legacy=0x7fa8400b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:42.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa837fff640 1 -- 172.21.15.18:0/3539397647 >> v1:172.21.15.18:6789/0 conn(0x7fa8400aedf0 legacy=0x7fa84014e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:42.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa837fff640 10 monclient: _finish_hunting 0 2024-03-25T17:58:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa837fff640 1 monclient: found mon.noname-c 2024-03-25T17:58:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa837fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:58:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa837fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T17:58:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa837fff640 1 -- 172.21.15.18:0/3539397647 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8400ae2c0 con 0x7fa8400b12d0 2024-03-25T17:58:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa837fff640 10 monclient: _finish_auth 0 2024-03-25T17:58:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa837fff640 10 monclient: _check_auth_tickets 2024-03-25T17:58:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa837fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:58:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa837fff640 1 -- 172.21.15.18:0/3539397647 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa830001690 con 0x7fa8400b12d0 2024-03-25T17:58:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa8477cf640 5 monclient: authenticate success, global_id 4315 2024-03-25T17:58:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa8477cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:58:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa837fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa837fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:58:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa837fff640 10 monclient: dump: 2024-03-25T17:58:42.650 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:42.650 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:42.650 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:42.650 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:42.650 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:42.650 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:42.650 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:42.650 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:42.650 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:42.650 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:42.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.647+0000 7fa8477cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:58:42.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.664+0000 7fa837fff640 1 -- 172.21.15.18:0/3539397647 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8300017f0 con 0x7fa8400b12d0 2024-03-25T17:58:42.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.664+0000 7fa837fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:58:42.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.664+0000 7fa837fff640 1 -- 172.21.15.18:0/3539397647 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa830001b80 con 0x7fa8400b12d0 2024-03-25T17:58:42.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.664+0000 7fa8477cf640 10 monclient: get_monmap_and_config success 2024-03-25T17:58:42.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.664+0000 7fa8477cf640 4 set_mon_vals no callback set 2024-03-25T17:58:42.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.664+0000 7fa837fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:42.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.664+0000 7fa837fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:58:42.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.664+0000 7fa837fff640 10 monclient: dump: 2024-03-25T17:58:42.665 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:42.666 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:42.666 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:42.666 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:42.666 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:42.666 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:42.666 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:42.666 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:42.666 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:42.666 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:42.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.665+0000 7fa8477cf640 10 monclient: shutdown 2024-03-25T17:58:42.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.665+0000 7fa8477cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:58:42.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.665+0000 7fa8477cf640 1 -- 172.21.15.18:0/3539397647 >> v1:172.21.15.169:6789/0 conn(0x7fa8400b12d0 legacy=0x7fa8400ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:42.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.665+0000 7fa8477cf640 1 -- 172.21.15.18:0/3539397647 shutdown_connections 2024-03-25T17:58:42.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.665+0000 7fa8477cf640 1 -- 172.21.15.18:0/3539397647 >> 172.21.15.18:0/3539397647 conn(0x7fa8400b5d10 msgr2=0x7fa8400b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:58:42.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.665+0000 7fa8477cf640 1 -- 172.21.15.18:0/3539397647 shutdown_connections 2024-03-25T17:58:42.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.665+0000 7fa8477cf640 1 -- 172.21.15.18:0/3539397647 wait complete. 2024-03-25T17:58:42.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.666+0000 7fa8477cf640 10 monclient: build_initial_monmap 2024-03-25T17:58:42.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.666+0000 7fa8477cf640 10 monclient: monmap: 2024-03-25T17:58:42.669 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:58:42.669 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:58:42.669 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:58:42.669 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:58:42.669 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:58:42.669 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:58:42.670 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:58:42.670 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:58:42.670 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:58:42.670 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:42.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.666+0000 7fa8477cf640 1 librados: starting msgr at 2024-03-25T17:58:42.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.666+0000 7fa8477cf640 1 librados: starting objecter 2024-03-25T17:58:42.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.666+0000 7fa8477cf640 1 Processor -- start 2024-03-25T17:58:42.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.666+0000 7fa8477cf640 1 -- start start 2024-03-25T17:58:42.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.666+0000 7fa8477cf640 1 librados: setting wanted keys 2024-03-25T17:58:42.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.666+0000 7fa8477cf640 1 librados: calling monclient init 2024-03-25T17:58:42.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.666+0000 7fa8477cf640 10 monclient: init 2024-03-25T17:58:42.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.667+0000 7fa8477cf640 10 monclient: _reopen_session rank -1 2024-03-25T17:58:42.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.667+0000 7fa8477cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:58:42.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.667+0000 7fa8477cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa84006c380 addr v1:172.21.15.169:6789/0 2024-03-25T17:58:42.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.667+0000 7fa8477cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa8400aedf0 addr v1:172.21.15.18:6790/0 2024-03-25T17:58:42.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.667+0000 7fa8477cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa8400b12d0 addr v1:172.21.15.18:6789/0 2024-03-25T17:58:42.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.667+0000 7fa8477cf640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa84006c8e0 con 0x7fa8400b12d0 2024-03-25T17:58:42.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.667+0000 7fa8477cf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8401c3550 con 0x7fa84006c380 2024-03-25T17:58:42.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.667+0000 7fa8477cf640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8401c1930 con 0x7fa8400aedf0 2024-03-25T17:58:42.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.667+0000 7fa8477cf640 10 monclient(hunting): _renew_subs 2024-03-25T17:58:42.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.667+0000 7fa8477cf640 10 monclient(hunting): authenticate will time out at 7051.246039s 2024-03-25T17:58:42.687 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.686+0000 7fa844d6a640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fa8400aedf0 0x7fa8401c20d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:39914/0 (socket says 172.21.15.18:39914) 2024-03-25T17:58:42.687 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.686+0000 7fa844d6a640 1 -- 172.21.15.18:0/2418322000 learned_addr learned my addr 172.21.15.18:0/2418322000 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:58:42.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.691+0000 7fa835ffb640 10 client.?.objecter ms_handle_connect 0x7fa8400aedf0 2024-03-25T17:58:42.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.691+0000 7fa835ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:42.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.691+0000 7fa835ffb640 10 client.?.objecter ms_handle_connect 0x7fa84006c380 2024-03-25T17:58:42.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.691+0000 7fa835ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:42.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.691+0000 7fa835ffb640 10 client.?.objecter ms_handle_connect 0x7fa8400b12d0 2024-03-25T17:58:42.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.691+0000 7fa835ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:42.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.701+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450461165 0 0) 0x7fa8401c1930 con 0x7fa8400aedf0 2024-03-25T17:58:42.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.701+0000 7fa835ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:58:42.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.701+0000 7fa835ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:42.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.701+0000 7fa835ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:42.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.701+0000 7fa835ffb640 10 monclient(hunting): my global_id is 4337 2024-03-25T17:58:42.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.701+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa824004eb0 con 0x7fa8400aedf0 2024-03-25T17:58:42.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.705+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346787620 0 0) 0x7fa8401c3550 con 0x7fa84006c380 2024-03-25T17:58:42.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.705+0000 7fa835ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:58:42.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.705+0000 7fa835ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:42.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.705+0000 7fa835ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:42.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.705+0000 7fa835ffb640 10 monclient(hunting): my global_id is 4318 2024-03-25T17:58:42.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.705+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8401c1930 con 0x7fa84006c380 2024-03-25T17:58:42.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.706+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907320579 0 0) 0x7fa84006c8e0 con 0x7fa8400b12d0 2024-03-25T17:58:42.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.706+0000 7fa835ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:58:42.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.706+0000 7fa835ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:42.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.706+0000 7fa835ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:42.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.706+0000 7fa835ffb640 10 monclient(hunting): my global_id is 4329 2024-03-25T17:58:42.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.706+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8401c3550 con 0x7fa8400b12d0 2024-03-25T17:58:42.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.712+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1706648609 0 0) 0x7fa824004eb0 con 0x7fa8400aedf0 2024-03-25T17:58:42.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.712+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa84006c8e0 con 0x7fa8400aedf0 2024-03-25T17:58:42.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.712+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa830001bc0 con 0x7fa8400aedf0 2024-03-25T17:58:42.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.712+0000 7fa835ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:42.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.712+0000 7fa835ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:58:42.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.712+0000 7fa835ffb640 10 monclient(hunting): dump: 2024-03-25T17:58:42.713 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:42.713 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:42.713 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:42.713 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:42.713 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:42.714 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:42.714 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:42.714 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:42.714 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:42.714 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:42.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.712+0000 7fa835ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:58:42.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.719+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 87121579 0 0) 0x7fa8401c1930 con 0x7fa84006c380 2024-03-25T17:58:42.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.719+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa824004eb0 con 0x7fa84006c380 2024-03-25T17:58:42.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.719+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa83c001970 con 0x7fa84006c380 2024-03-25T17:58:42.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.719+0000 7fa835ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:42.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.719+0000 7fa835ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:58:42.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.719+0000 7fa835ffb640 10 monclient(hunting): dump: 2024-03-25T17:58:42.720 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:42.720 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:42.720 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:42.721 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:42.721 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:42.721 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:42.721 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:42.721 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:42.721 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:42.721 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:42.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.725+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1502633797 0 0) 0x7fa8401c3550 con 0x7fa8400b12d0 2024-03-25T17:58:42.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.725+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8401c1930 con 0x7fa8400b12d0 2024-03-25T17:58:42.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.725+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa838001c70 con 0x7fa8400b12d0 2024-03-25T17:58:42.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.725+0000 7fa835ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:42.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.725+0000 7fa835ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:58:42.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.725+0000 7fa835ffb640 10 monclient(hunting): dump: 2024-03-25T17:58:42.726 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:42.726 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:42.726 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:42.726 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:42.726 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:42.726 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:42.726 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:42.726 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:42.726 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:42.726 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:42.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 506354810 0 0) 0x7fa84006c8e0 con 0x7fa8400aedf0 2024-03-25T17:58:42.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 >> v1:172.21.15.169:6789/0 conn(0x7fa84006c380 legacy=0x7fa8401db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:42.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 >> v1:172.21.15.18:6789/0 conn(0x7fa8400b12d0 legacy=0x7fa8401c27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:42.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa835ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:58:42.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa835ffb640 1 monclient: found mon.c 2024-03-25T17:58:42.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa835ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:58:42.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa835ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:58:42.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8401c1ac0 con 0x7fa8400aedf0 2024-03-25T17:58:42.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa835ffb640 10 monclient: _finish_auth 0 2024-03-25T17:58:42.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa835ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:58:42.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa835ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:58:42.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa8477cf640 5 monclient: authenticate success, global_id 4337 2024-03-25T17:58:42.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa8477cf640 10 monclient: _renew_subs 2024-03-25T17:58:42.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa8477cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:58:42.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa8477cf640 1 -- 172.21.15.18:0/2418322000 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8401c4ca0 con 0x7fa8400aedf0 2024-03-25T17:58:42.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa8477cf640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:58:42.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa8477cf640 10 monclient: _renew_subs 2024-03-25T17:58:42.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa8477cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:58:42.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa8477cf640 1 -- 172.21.15.18:0/2418322000 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8401c2ef0 con 0x7fa8400aedf0 2024-03-25T17:58:42.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.727+0000 7fa8477cf640 1 librados: init done 2024-03-25T17:58:42.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.728+0000 7fa8477cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:42.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.728+0000 7fa8477cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:42.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.728+0000 7fa8477cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:58:42.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.728+0000 7fa8477cf640 1 -- 172.21.15.18:0/2418322000 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8400627b0 con 0x7fa8400aedf0 2024-03-25T17:58:42.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.753+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8300011f0 con 0x7fa8400aedf0 2024-03-25T17:58:42.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.753+0000 7fa835ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:58:42.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.753+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa830001d80 con 0x7fa8400aedf0 2024-03-25T17:58:42.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.753+0000 7fa835ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:42.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.753+0000 7fa835ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:58:42.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.753+0000 7fa835ffb640 10 monclient: dump: 2024-03-25T17:58:42.754 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:42.754 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:42.754 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:42.754 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:42.754 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:42.755 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:42.755 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:42.755 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:42.755 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:42.755 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:42.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.753+0000 7fa8377fe640 4 set_mon_vals no callback set 2024-03-25T17:58:42.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.754+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fa83001ab20 con 0x7fa8400aedf0 2024-03-25T17:58:42.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.754+0000 7fa84556b640 10 monclient: get_auth_request con 0x7fa82406e0b0 auth_method 0 2024-03-25T17:58:42.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:42.755+0000 7fa835ffb640 10 client.4337.objecter ms_handle_connect 0x7fa82406e0b0 2024-03-25T17:58:43.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.048+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fa830054860 con 0x7fa8400aedf0 2024-03-25T17:58:43.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.048+0000 7fa835ffb640 10 client.4337.objecter ms_dispatch 0x7fa840000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:58:43.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.048+0000 7fa835ffb640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:58:43.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.048+0000 7fa835ffb640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:58:43.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.048+0000 7fa835ffb640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-25T17:58:43.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.048+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa8300501a0 con 0x7fa8400aedf0 2024-03-25T17:58:43.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.048+0000 7fa835ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:43.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.048+0000 7fa835ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:58:43.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.313+0000 7fa8477cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:43.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.313+0000 7fa8477cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:43.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.313+0000 7fa8477cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:58:43.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.313+0000 7fa8477cf640 1 -- 172.21.15.18:0/2418322000 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa840069f40 con 0x7fa8400aedf0 2024-03-25T17:58:43.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.336+0000 7fa835ffb640 1 -- 172.21.15.18:0/2418322000 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fa8300522a0 con 0x7fa8400aedf0 2024-03-25T17:58:43.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.336+0000 7fa835ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:43.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.336+0000 7fa835ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:58:43.337 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:58:43.338 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:58:43.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.338+0000 7fa8477cf640 10 librados: watch_flush enter 2024-03-25T17:58:43.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.338+0000 7fa8477cf640 10 librados: watch_flush exit 2024-03-25T17:58:43.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.338+0000 7fa8477cf640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-25T17:58:43.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.338+0000 7fa8477cf640 10 client.4337.objecter successfully canceled tick 2024-03-25T17:58:43.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.339+0000 7fa8477cf640 1 -- 172.21.15.18:0/2418322000 >> v1:172.21.15.18:6800/3721244451 conn(0x7fa82406e0b0 legacy=0x7fa824070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:43.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.339+0000 7fa8477cf640 10 monclient: shutdown 2024-03-25T17:58:43.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.339+0000 7fa8477cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:58:43.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.339+0000 7fa8477cf640 1 -- 172.21.15.18:0/2418322000 >> v1:172.21.15.18:6790/0 conn(0x7fa8400aedf0 legacy=0x7fa8401c20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:43.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.339+0000 7fa8477cf640 1 -- 172.21.15.18:0/2418322000 shutdown_connections 2024-03-25T17:58:43.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.339+0000 7fa8477cf640 1 -- 172.21.15.18:0/2418322000 >> 172.21.15.18:0/2418322000 conn(0x7fa8400b5d10 msgr2=0x7fa8400ae680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:58:43.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.339+0000 7fa8477cf640 1 -- 172.21.15.18:0/2418322000 shutdown_connections 2024-03-25T17:58:43.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.339+0000 7fa8477cf640 1 -- 172.21.15.18:0/2418322000 wait complete. 2024-03-25T17:58:43.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:43.339+0000 7fa8477cf640 1 librados: shutdown 2024-03-25T17:58:43.353 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:58:50.159 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:58:50.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.276+0000 7fcc2384e640 10 monclient: get_monmap_and_config 2024-03-25T17:58:50.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.276+0000 7fcc2384e640 10 monclient: build_initial_monmap 2024-03-25T17:58:50.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.276+0000 7fcc2384e640 10 monclient: monmap: 2024-03-25T17:58:50.277 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:58:50.277 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:50.277 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:58:50.277691+0000 2024-03-25T17:58:50.278 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:58:50.277691+0000 2024-03-25T17:58:50.278 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:58:50.278 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:50.278 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:58:50.278 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:58:50.278 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:58:50.278 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:50.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.277+0000 7fcc2384e640 1 Processor -- start 2024-03-25T17:58:50.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.277+0000 7fcc2384e640 1 -- start start 2024-03-25T17:58:50.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.277+0000 7fcc2384e640 10 monclient: init 2024-03-25T17:58:50.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.277+0000 7fcc2384e640 10 monclient: _reopen_session rank -1 2024-03-25T17:58:50.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.278+0000 7fcc2384e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:58:50.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.278+0000 7fcc2384e640 10 monclient(hunting): picked mon.noname-a con 0x7fcc1c0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T17:58:50.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.278+0000 7fcc2384e640 10 monclient(hunting): picked mon.noname-c con 0x7fcc1c13f710 addr v1:172.21.15.169:6789/0 2024-03-25T17:58:50.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.278+0000 7fcc2384e640 10 monclient(hunting): picked mon.noname-b con 0x7fcc1c1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T17:58:50.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.278+0000 7fcc2384e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1c142520 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.278+0000 7fcc2384e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1c0b28a0 con 0x7fcc1c13f710 2024-03-25T17:58:50.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.278+0000 7fcc2384e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1c0b2a30 con 0x7fcc1c1430c0 2024-03-25T17:58:50.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.278+0000 7fcc2384e640 10 monclient(hunting): _renew_subs 2024-03-25T17:58:50.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.278+0000 7fcc2384e640 10 monclient(hunting): authenticate will time out at 7058.856839s 2024-03-25T17:58:50.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.283+0000 7fcc21deb640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fcc1c1430c0 0x7fcc1c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:37620/0 (socket says 172.21.15.18:37620) 2024-03-25T17:58:50.284 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.283+0000 7fcc21deb640 1 -- 172.21.15.18:0/1233559179 learned_addr learned my addr 172.21.15.18:0/1233559179 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:58:50.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.306+0000 7fcc13fff640 1 -- 172.21.15.18:0/1233559179 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443899819 0 0) 0x7fcc1c0b2a30 con 0x7fcc1c1430c0 2024-03-25T17:58:50.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.306+0000 7fcc13fff640 10 monclient(hunting): _negotiate 2024-03-25T17:58:50.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.306+0000 7fcc13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:50.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.306+0000 7fcc13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:50.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.306+0000 7fcc13fff640 10 monclient(hunting): my global_id is 4340 2024-03-25T17:58:50.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.306+0000 7fcc13fff640 1 -- 172.21.15.18:0/1233559179 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc08003520 con 0x7fcc1c1430c0 2024-03-25T17:58:50.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.307+0000 7fcc13fff640 1 -- 172.21.15.18:0/1233559179 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235123781 0 0) 0x7fcc1c0b28a0 con 0x7fcc1c13f710 2024-03-25T17:58:50.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.307+0000 7fcc13fff640 10 monclient(hunting): _negotiate 2024-03-25T17:58:50.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.307+0000 7fcc13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:50.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.307+0000 7fcc13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:50.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.307+0000 7fcc13fff640 10 monclient(hunting): my global_id is 4321 2024-03-25T17:58:50.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.307+0000 7fcc13fff640 1 -- 172.21.15.18:0/1233559179 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc1c0b2a30 con 0x7fcc1c13f710 2024-03-25T17:58:50.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.315+0000 7fcc13fff640 1 -- 172.21.15.18:0/1233559179 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2635093519 0 0) 0x7fcc08003520 con 0x7fcc1c1430c0 2024-03-25T17:58:50.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.315+0000 7fcc13fff640 1 -- 172.21.15.18:0/1233559179 >> v1:172.21.15.169:6789/0 conn(0x7fcc1c13f710 legacy=0x7fcc1c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.315+0000 7fcc13fff640 1 -- 172.21.15.18:0/1233559179 >> v1:172.21.15.18:6789/0 conn(0x7fcc1c0a98f0 legacy=0x7fcc1c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.315+0000 7fcc13fff640 10 monclient: _finish_hunting 0 2024-03-25T17:58:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.315+0000 7fcc13fff640 1 monclient: found mon.noname-b 2024-03-25T17:58:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.315+0000 7fcc13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:58:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.315+0000 7fcc13fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T17:58:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.315+0000 7fcc13fff640 1 -- 172.21.15.18:0/1233559179 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c0b44c0 con 0x7fcc1c1430c0 2024-03-25T17:58:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.315+0000 7fcc13fff640 10 monclient: _finish_auth 0 2024-03-25T17:58:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.315+0000 7fcc13fff640 10 monclient: _check_auth_tickets 2024-03-25T17:58:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.315+0000 7fcc13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:58:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.315+0000 7fcc13fff640 1 -- 172.21.15.18:0/1233559179 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fcc14001a10 con 0x7fcc1c1430c0 2024-03-25T17:58:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.315+0000 7fcc13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.315+0000 7fcc13fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:58:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.315+0000 7fcc13fff640 10 monclient: dump: 2024-03-25T17:58:50.317 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:50.318 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:50.318 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:50.318 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:50.318 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:50.318 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:50.318 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:50.318 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:50.318 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:50.318 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:50.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.315+0000 7fcc2384e640 5 monclient: authenticate success, global_id 4340 2024-03-25T17:58:50.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.315+0000 7fcc2384e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:58:50.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.333+0000 7fcc13fff640 1 -- 172.21.15.18:0/1233559179 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc14001b70 con 0x7fcc1c1430c0 2024-03-25T17:58:50.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.333+0000 7fcc13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:58:50.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.333+0000 7fcc13fff640 1 -- 172.21.15.18:0/1233559179 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fcc14001f80 con 0x7fcc1c1430c0 2024-03-25T17:58:50.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.333+0000 7fcc13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:50.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.333+0000 7fcc13fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:58:50.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.333+0000 7fcc13fff640 10 monclient: dump: 2024-03-25T17:58:50.334 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:50.334 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:50.334 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:50.334 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:50.335 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:50.335 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:50.335 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:50.335 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:50.335 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:50.335 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:50.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.333+0000 7fcc2384e640 10 monclient: get_monmap_and_config success 2024-03-25T17:58:50.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.333+0000 7fcc2384e640 4 set_mon_vals no callback set 2024-03-25T17:58:50.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.334+0000 7fcc2384e640 10 monclient: shutdown 2024-03-25T17:58:50.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.334+0000 7fcc2384e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:58:50.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.334+0000 7fcc2384e640 1 -- 172.21.15.18:0/1233559179 >> v1:172.21.15.18:6790/0 conn(0x7fcc1c1430c0 legacy=0x7fcc1c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:50.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.334+0000 7fcc2384e640 1 -- 172.21.15.18:0/1233559179 shutdown_connections 2024-03-25T17:58:50.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.334+0000 7fcc2384e640 1 -- 172.21.15.18:0/1233559179 >> 172.21.15.18:0/1233559179 conn(0x7fcc1c0b7ea0 msgr2=0x7fcc1c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:58:50.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.334+0000 7fcc2384e640 1 -- 172.21.15.18:0/1233559179 shutdown_connections 2024-03-25T17:58:50.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.334+0000 7fcc2384e640 1 -- 172.21.15.18:0/1233559179 wait complete. 2024-03-25T17:58:50.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.334+0000 7fcc2384e640 10 monclient: build_initial_monmap 2024-03-25T17:58:50.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.335+0000 7fcc2384e640 10 monclient: monmap: 2024-03-25T17:58:50.337 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:58:50.337 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:58:50.337 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:58:50.337 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:58:50.338 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:58:50.338 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:58:50.338 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:58:50.338 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:58:50.338 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:58:50.338 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:50.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.335+0000 7fcc2384e640 1 librados: starting msgr at 2024-03-25T17:58:50.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.335+0000 7fcc2384e640 1 librados: starting objecter 2024-03-25T17:58:50.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.335+0000 7fcc2384e640 1 Processor -- start 2024-03-25T17:58:50.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.335+0000 7fcc2384e640 1 -- start start 2024-03-25T17:58:50.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.335+0000 7fcc2384e640 1 librados: setting wanted keys 2024-03-25T17:58:50.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.335+0000 7fcc2384e640 1 librados: calling monclient init 2024-03-25T17:58:50.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.335+0000 7fcc2384e640 10 monclient: init 2024-03-25T17:58:50.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.336+0000 7fcc2384e640 10 monclient: _reopen_session rank -1 2024-03-25T17:58:50.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.336+0000 7fcc2384e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:58:50.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.336+0000 7fcc2384e640 10 monclient(hunting): picked mon.noname-b con 0x7fcc1c0a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T17:58:50.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.336+0000 7fcc2384e640 10 monclient(hunting): picked mon.noname-c con 0x7fcc1c13f710 addr v1:172.21.15.18:6790/0 2024-03-25T17:58:50.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.336+0000 7fcc2384e640 10 monclient(hunting): picked mon.noname-a con 0x7fcc1c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T17:58:50.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.336+0000 7fcc2384e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1c0abdc0 con 0x7fcc1c1430c0 2024-03-25T17:58:50.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.336+0000 7fcc2384e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1c1d7730 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.336+0000 7fcc2384e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1c138f10 con 0x7fcc1c13f710 2024-03-25T17:58:50.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.336+0000 7fcc2384e640 10 monclient(hunting): _renew_subs 2024-03-25T17:58:50.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.336+0000 7fcc2384e640 10 monclient(hunting): authenticate will time out at 7058.915129s 2024-03-25T17:58:50.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.342+0000 7fcc215ea640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fcc1c0a98f0 0x7fcc1c138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:39514/0 (socket says 172.21.15.18:39514) 2024-03-25T17:58:50.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.342+0000 7fcc215ea640 1 -- 172.21.15.18:0/775893863 learned_addr learned my addr 172.21.15.18:0/775893863 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:58:50.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.344+0000 7fcc11ffb640 10 client.?.objecter ms_handle_connect 0x7fcc1c0a98f0 2024-03-25T17:58:50.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.344+0000 7fcc11ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:50.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.352+0000 7fcc11ffb640 1 -- 172.21.15.18:0/775893863 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354296508 0 0) 0x7fcc1c1d7730 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.352+0000 7fcc11ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:58:50.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.352+0000 7fcc11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:50.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.352+0000 7fcc11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:50.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.352+0000 7fcc11ffb640 10 monclient(hunting): my global_id is 4324 2024-03-25T17:58:50.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.352+0000 7fcc11ffb640 1 -- 172.21.15.18:0/775893863 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc00004eb0 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.352+0000 7fcc11ffb640 10 client.?.objecter ms_handle_connect 0x7fcc1c13f710 2024-03-25T17:58:50.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.352+0000 7fcc11ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:50.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.353+0000 7fcc11ffb640 10 client.?.objecter ms_handle_connect 0x7fcc1c1430c0 2024-03-25T17:58:50.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.353+0000 7fcc11ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:50.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.359+0000 7fcc11ffb640 1 -- 172.21.15.18:0/775893863 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554870361 0 0) 0x7fcc1c0abdc0 con 0x7fcc1c1430c0 2024-03-25T17:58:50.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.359+0000 7fcc11ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:58:50.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.359+0000 7fcc11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:50.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.359+0000 7fcc11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:50.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.359+0000 7fcc11ffb640 10 monclient(hunting): my global_id is 4335 2024-03-25T17:58:50.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.360+0000 7fcc11ffb640 1 -- 172.21.15.18:0/775893863 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc1c1d7730 con 0x7fcc1c1430c0 2024-03-25T17:58:50.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.361+0000 7fcc11ffb640 1 -- 172.21.15.18:0/775893863 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1914345746 0 0) 0x7fcc00004eb0 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.361+0000 7fcc11ffb640 1 -- 172.21.15.18:0/775893863 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc1c0abdc0 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.361+0000 7fcc11ffb640 1 -- 172.21.15.18:0/775893863 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fcc180019d0 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.361+0000 7fcc11ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:50.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.361+0000 7fcc11ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:58:50.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.361+0000 7fcc11ffb640 10 monclient(hunting): dump: 2024-03-25T17:58:50.362 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:50.362 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:50.362 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:50.362 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:50.362 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:50.363 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:50.363 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:50.363 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:50.363 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:50.363 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:50.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.361+0000 7fcc11ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:58:50.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc11ffb640 1 -- 172.21.15.18:0/775893863 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2670180628 0 0) 0x7fcc1c0abdc0 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc11ffb640 1 -- 172.21.15.18:0/775893863 >> v1:172.21.15.18:6790/0 conn(0x7fcc1c13f710 legacy=0x7fcc1c1d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:50.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc11ffb640 1 -- 172.21.15.18:0/775893863 >> v1:172.21.15.18:6789/0 conn(0x7fcc1c1430c0 legacy=0x7fcc1c1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:50.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc11ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:58:50.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc11ffb640 1 monclient: found mon.b 2024-03-25T17:58:50.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:58:50.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc11ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:58:50.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc11ffb640 1 -- 172.21.15.18:0/775893863 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c1390a0 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc11ffb640 10 monclient: _finish_auth 0 2024-03-25T17:58:50.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc11ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:58:50.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:58:50.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc2384e640 5 monclient: authenticate success, global_id 4324 2024-03-25T17:58:50.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc2384e640 10 monclient: _renew_subs 2024-03-25T17:58:50.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc2384e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:58:50.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc2384e640 1 -- 172.21.15.18:0/775893863 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1c1d8320 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc2384e640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:58:50.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc2384e640 10 monclient: _renew_subs 2024-03-25T17:58:50.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc2384e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:58:50.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc2384e640 1 -- 172.21.15.18:0/775893863 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc1c1d43c0 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.366+0000 7fcc2384e640 1 librados: init done 2024-03-25T17:58:50.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.367+0000 7fcc2384e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:50.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.367+0000 7fcc2384e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:50.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.367+0000 7fcc2384e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:58:50.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.367+0000 7fcc2384e640 1 -- 172.21.15.18:0/775893863 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc1c0627b0 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.643+0000 7fcc11ffb640 1 -- 172.21.15.18:0/775893863 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc180035f0 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.643+0000 7fcc11ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:58:50.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.643+0000 7fcc11ffb640 1 -- 172.21.15.18:0/775893863 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fcc18003980 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.643+0000 7fcc11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:50.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.643+0000 7fcc11ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:58:50.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.643+0000 7fcc11ffb640 10 monclient: dump: 2024-03-25T17:58:50.645 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:50.645 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:50.645 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:50.645 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:50.645 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:50.645 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:50.645 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:50.645 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:50.645 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:50.646 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:50.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.643+0000 7fcc137fe640 4 set_mon_vals no callback set 2024-03-25T17:58:50.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.644+0000 7fcc11ffb640 1 -- 172.21.15.18:0/775893863 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fcc18003ba0 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.645+0000 7fcc20de9640 10 monclient: get_auth_request con 0x7fcc0006dd10 auth_method 0 2024-03-25T17:58:50.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.645+0000 7fcc11ffb640 10 client.4324.objecter ms_handle_connect 0x7fcc0006dd10 2024-03-25T17:58:50.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.649+0000 7fcc11ffb640 1 -- 172.21.15.18:0/775893863 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fcc180547c0 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.649+0000 7fcc11ffb640 10 client.4324.objecter ms_dispatch 0x7fcc1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:58:50.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.650+0000 7fcc11ffb640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:58:50.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.650+0000 7fcc11ffb640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:58:50.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.650+0000 7fcc11ffb640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-25T17:58:50.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.650+0000 7fcc11ffb640 1 -- 172.21.15.18:0/775893863 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc180502d0 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.650+0000 7fcc11ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:50.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.650+0000 7fcc11ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:58:50.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.909+0000 7fcc2384e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:50.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.910+0000 7fcc2384e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:50.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.910+0000 7fcc2384e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:58:50.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.910+0000 7fcc2384e640 1 -- 172.21.15.18:0/775893863 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc1c069f40 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.933+0000 7fcc11ffb640 1 -- 172.21.15.18:0/775893863 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fcc18052200 con 0x7fcc1c0a98f0 2024-03-25T17:58:50.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.933+0000 7fcc11ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:50.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.933+0000 7fcc11ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:58:50.935 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:58:50.935 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:58:50.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.935+0000 7fcc2384e640 10 librados: watch_flush enter 2024-03-25T17:58:50.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.935+0000 7fcc2384e640 10 librados: watch_flush exit 2024-03-25T17:58:50.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.935+0000 7fcc2384e640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-25T17:58:50.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.935+0000 7fcc2384e640 10 client.4324.objecter successfully canceled tick 2024-03-25T17:58:50.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.935+0000 7fcc2384e640 1 -- 172.21.15.18:0/775893863 >> v1:172.21.15.18:6800/3721244451 conn(0x7fcc0006dd10 legacy=0x7fcc000701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:50.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.935+0000 7fcc2384e640 10 monclient: shutdown 2024-03-25T17:58:50.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.935+0000 7fcc2384e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:58:50.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.935+0000 7fcc2384e640 1 -- 172.21.15.18:0/775893863 >> v1:172.21.15.169:6789/0 conn(0x7fcc1c0a98f0 legacy=0x7fcc1c138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:50.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.935+0000 7fcc2384e640 1 -- 172.21.15.18:0/775893863 shutdown_connections 2024-03-25T17:58:50.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.935+0000 7fcc2384e640 1 -- 172.21.15.18:0/775893863 >> 172.21.15.18:0/775893863 conn(0x7fcc1c0b7ea0 msgr2=0x7fcc1c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:58:50.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.935+0000 7fcc2384e640 1 -- 172.21.15.18:0/775893863 shutdown_connections 2024-03-25T17:58:50.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.935+0000 7fcc2384e640 1 -- 172.21.15.18:0/775893863 wait complete. 2024-03-25T17:58:50.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:50.936+0000 7fcc2384e640 1 librados: shutdown 2024-03-25T17:58:50.947 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:58:57.755 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:58:57.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.868+0000 7f1c31255640 10 monclient: get_monmap_and_config 2024-03-25T17:58:57.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.868+0000 7f1c31255640 10 monclient: build_initial_monmap 2024-03-25T17:58:57.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.868+0000 7f1c31255640 10 monclient: monmap: 2024-03-25T17:58:57.869 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:58:57.869 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:57.869 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:58:57.869680+0000 2024-03-25T17:58:57.869 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:58:57.869680+0000 2024-03-25T17:58:57.869 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:58:57.869 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:57.870 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:58:57.870 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:58:57.870 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:58:57.870 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:57.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.869+0000 7f1c31255640 1 Processor -- start 2024-03-25T17:58:57.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.869+0000 7f1c31255640 1 -- start start 2024-03-25T17:58:57.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.869+0000 7f1c31255640 10 monclient: init 2024-03-25T17:58:57.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.869+0000 7f1c31255640 10 monclient: _reopen_session rank -1 2024-03-25T17:58:57.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.869+0000 7f1c31255640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:58:57.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.869+0000 7f1c31255640 10 monclient(hunting): picked mon.noname-c con 0x7f1c2c0a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T17:58:57.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.869+0000 7f1c31255640 10 monclient(hunting): picked mon.noname-b con 0x7f1c2c13f710 addr v1:172.21.15.18:6790/0 2024-03-25T17:58:57.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.869+0000 7f1c31255640 10 monclient(hunting): picked mon.noname-a con 0x7f1c2c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T17:58:57.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.869+0000 7f1c31255640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c142520 con 0x7f1c2c1430c0 2024-03-25T17:58:57.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.870+0000 7f1c31255640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c0b28a0 con 0x7f1c2c0a98f0 2024-03-25T17:58:57.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.870+0000 7f1c31255640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c0b2a30 con 0x7f1c2c13f710 2024-03-25T17:58:57.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.870+0000 7f1c31255640 10 monclient(hunting): _renew_subs 2024-03-25T17:58:57.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.870+0000 7f1c31255640 10 monclient(hunting): authenticate will time out at 7066.448771s 2024-03-25T17:58:57.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.876+0000 7f1c2b59e640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f1c2c1430c0 0x7f1c2c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:51512/0 (socket says 172.21.15.18:51512) 2024-03-25T17:58:57.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.876+0000 7f1c2b59e640 1 -- 172.21.15.18:0/836503135 learned_addr learned my addr 172.21.15.18:0/836503135 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:58:57.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.886+0000 7f1c29d9b640 1 -- 172.21.15.18:0/836503135 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3241609131 0 0) 0x7f1c2c142520 con 0x7f1c2c1430c0 2024-03-25T17:58:57.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.886+0000 7f1c29d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:58:57.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.886+0000 7f1c29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:57.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.886+0000 7f1c29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:57.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.886+0000 7f1c29d9b640 10 monclient(hunting): my global_id is 4338 2024-03-25T17:58:57.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.886+0000 7f1c29d9b640 1 -- 172.21.15.18:0/836503135 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c14003520 con 0x7f1c2c1430c0 2024-03-25T17:58:57.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.893+0000 7f1c29d9b640 1 -- 172.21.15.18:0/836503135 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 218994976 0 0) 0x7f1c2c0b2a30 con 0x7f1c2c13f710 2024-03-25T17:58:57.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.893+0000 7f1c29d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:58:57.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.893+0000 7f1c29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:57.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.893+0000 7f1c29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:57.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.893+0000 7f1c29d9b640 10 monclient(hunting): my global_id is 4346 2024-03-25T17:58:57.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.893+0000 7f1c29d9b640 1 -- 172.21.15.18:0/836503135 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c2c142520 con 0x7f1c2c13f710 2024-03-25T17:58:57.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.895+0000 7f1c29d9b640 1 -- 172.21.15.18:0/836503135 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 688886479 0 0) 0x7f1c14003520 con 0x7f1c2c1430c0 2024-03-25T17:58:57.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.895+0000 7f1c29d9b640 1 -- 172.21.15.18:0/836503135 >> v1:172.21.15.18:6790/0 conn(0x7f1c2c13f710 legacy=0x7f1c2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:57.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.895+0000 7f1c29d9b640 1 -- 172.21.15.18:0/836503135 >> v1:172.21.15.169:6789/0 conn(0x7f1c2c0a98f0 legacy=0x7f1c2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:57.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.895+0000 7f1c29d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:58:57.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.895+0000 7f1c29d9b640 1 monclient: found mon.noname-a 2024-03-25T17:58:57.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.895+0000 7f1c29d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:58:57.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.895+0000 7f1c29d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T17:58:57.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.895+0000 7f1c29d9b640 1 -- 172.21.15.18:0/836503135 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c0b44c0 con 0x7f1c2c1430c0 2024-03-25T17:58:57.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.895+0000 7f1c29d9b640 10 monclient: _finish_auth 0 2024-03-25T17:58:57.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.895+0000 7f1c29d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:58:57.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.895+0000 7f1c29d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:58:57.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.895+0000 7f1c29d9b640 1 -- 172.21.15.18:0/836503135 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f1c24001a10 con 0x7f1c2c1430c0 2024-03-25T17:58:57.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.895+0000 7f1c29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:57.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.895+0000 7f1c29d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:58:57.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.895+0000 7f1c29d9b640 10 monclient: dump: 2024-03-25T17:58:57.897 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:57.897 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:57.897 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:57.898 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:57.898 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:57.898 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:57.898 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:57.898 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:57.898 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:57.898 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:57.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.895+0000 7f1c31255640 5 monclient: authenticate success, global_id 4338 2024-03-25T17:58:57.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.895+0000 7f1c31255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:58:57.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.906+0000 7f1c29d9b640 1 -- 172.21.15.18:0/836503135 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c24001b70 con 0x7f1c2c1430c0 2024-03-25T17:58:57.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.906+0000 7f1c29d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:58:57.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.906+0000 7f1c29d9b640 1 -- 172.21.15.18:0/836503135 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f1c24001f80 con 0x7f1c2c1430c0 2024-03-25T17:58:57.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.906+0000 7f1c29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:57.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.906+0000 7f1c29d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:58:57.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.906+0000 7f1c29d9b640 10 monclient: dump: 2024-03-25T17:58:57.907 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:57.907 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:57.907 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:57.907 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:57.907 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:57.907 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:57.907 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:57.907 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:57.908 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:57.908 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:57.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.906+0000 7f1c31255640 10 monclient: get_monmap_and_config success 2024-03-25T17:58:57.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.906+0000 7f1c31255640 4 set_mon_vals no callback set 2024-03-25T17:58:57.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.907+0000 7f1c31255640 10 monclient: shutdown 2024-03-25T17:58:57.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.907+0000 7f1c31255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:58:57.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.907+0000 7f1c31255640 1 -- 172.21.15.18:0/836503135 >> v1:172.21.15.18:6789/0 conn(0x7f1c2c1430c0 legacy=0x7f1c2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:57.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.907+0000 7f1c31255640 1 -- 172.21.15.18:0/836503135 shutdown_connections 2024-03-25T17:58:57.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.907+0000 7f1c31255640 1 -- 172.21.15.18:0/836503135 >> 172.21.15.18:0/836503135 conn(0x7f1c2c0b7ea0 msgr2=0x7f1c2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:58:57.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.907+0000 7f1c31255640 1 -- 172.21.15.18:0/836503135 shutdown_connections 2024-03-25T17:58:57.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.907+0000 7f1c31255640 1 -- 172.21.15.18:0/836503135 wait complete. 2024-03-25T17:58:57.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.908+0000 7f1c31255640 10 monclient: build_initial_monmap 2024-03-25T17:58:57.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.908+0000 7f1c31255640 10 monclient: monmap: 2024-03-25T17:58:57.911 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:58:57.911 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:58:57.911 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:58:57.911 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:58:57.911 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:58:57.911 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:58:57.911 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:58:57.911 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:58:57.911 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:58:57.911 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:57.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.908+0000 7f1c31255640 1 librados: starting msgr at 2024-03-25T17:58:57.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.908+0000 7f1c31255640 1 librados: starting objecter 2024-03-25T17:58:57.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.908+0000 7f1c31255640 1 Processor -- start 2024-03-25T17:58:57.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.908+0000 7f1c31255640 1 -- start start 2024-03-25T17:58:57.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.908+0000 7f1c31255640 1 librados: setting wanted keys 2024-03-25T17:58:57.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.908+0000 7f1c31255640 1 librados: calling monclient init 2024-03-25T17:58:57.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.908+0000 7f1c31255640 10 monclient: init 2024-03-25T17:58:57.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.909+0000 7f1c31255640 10 monclient: _reopen_session rank -1 2024-03-25T17:58:57.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.909+0000 7f1c31255640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:58:57.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.909+0000 7f1c31255640 10 monclient(hunting): picked mon.noname-c con 0x7f1c2c0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T17:58:57.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.909+0000 7f1c31255640 10 monclient(hunting): picked mon.noname-b con 0x7f1c2c13f710 addr v1:172.21.15.169:6789/0 2024-03-25T17:58:57.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.909+0000 7f1c31255640 10 monclient(hunting): picked mon.noname-a con 0x7f1c2c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T17:58:57.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.909+0000 7f1c31255640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c136e20 con 0x7f1c2c1430c0 2024-03-25T17:58:57.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.909+0000 7f1c31255640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c0b7940 con 0x7f1c2c13f710 2024-03-25T17:58:57.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.909+0000 7f1c31255640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c1351a0 con 0x7f1c2c0a98f0 2024-03-25T17:58:57.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.909+0000 7f1c31255640 10 monclient(hunting): _renew_subs 2024-03-25T17:58:57.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.909+0000 7f1c31255640 10 monclient(hunting): authenticate will time out at 7066.488007s 2024-03-25T17:58:57.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.920+0000 7f1c2ad9d640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f1c2c0a98f0 0x7f1c2c1daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:38564/0 (socket says 172.21.15.18:38564) 2024-03-25T17:58:57.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.921+0000 7f1c2ad9d640 1 -- 172.21.15.18:0/1691791070 learned_addr learned my addr 172.21.15.18:0/1691791070 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:58:57.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.922+0000 7f1c137fe640 10 client.?.objecter ms_handle_connect 0x7f1c2c0a98f0 2024-03-25T17:58:57.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.922+0000 7f1c137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:57.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.925+0000 7f1c137fe640 10 client.?.objecter ms_handle_connect 0x7f1c2c13f710 2024-03-25T17:58:57.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.925+0000 7f1c137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:57.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.926+0000 7f1c137fe640 10 client.?.objecter ms_handle_connect 0x7f1c2c1430c0 2024-03-25T17:58:57.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.926+0000 7f1c137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:58:57.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.932+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414154805 0 0) 0x7f1c2c1351a0 con 0x7f1c2c0a98f0 2024-03-25T17:58:57.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.932+0000 7f1c137fe640 10 monclient(hunting): _negotiate 2024-03-25T17:58:57.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.932+0000 7f1c137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:57.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.932+0000 7f1c137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:57.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.932+0000 7f1c137fe640 10 monclient(hunting): my global_id is 4349 2024-03-25T17:58:57.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.932+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c18004eb0 con 0x7f1c2c0a98f0 2024-03-25T17:58:57.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.934+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675000580 0 0) 0x7f1c2c0b7940 con 0x7f1c2c13f710 2024-03-25T17:58:57.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.934+0000 7f1c137fe640 10 monclient(hunting): _negotiate 2024-03-25T17:58:57.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.934+0000 7f1c137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:57.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.934+0000 7f1c137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:57.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.934+0000 7f1c137fe640 10 monclient(hunting): my global_id is 4330 2024-03-25T17:58:57.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.934+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c2c1351a0 con 0x7f1c2c13f710 2024-03-25T17:58:57.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.949+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520450157 0 0) 0x7f1c2c136e20 con 0x7f1c2c1430c0 2024-03-25T17:58:57.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.949+0000 7f1c137fe640 10 monclient(hunting): _negotiate 2024-03-25T17:58:57.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.949+0000 7f1c137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:58:57.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.949+0000 7f1c137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:58:57.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.949+0000 7f1c137fe640 10 monclient(hunting): my global_id is 4341 2024-03-25T17:58:57.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.949+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c2c0b7940 con 0x7f1c2c1430c0 2024-03-25T17:58:57.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.957+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3928484539 0 0) 0x7f1c2c1351a0 con 0x7f1c2c13f710 2024-03-25T17:58:57.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.957+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c2c136e20 con 0x7f1c2c13f710 2024-03-25T17:58:57.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.957+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f1c20001930 con 0x7f1c2c13f710 2024-03-25T17:58:57.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.958+0000 7f1c137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:57.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.958+0000 7f1c137fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:58:57.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.958+0000 7f1c137fe640 10 monclient(hunting): dump: 2024-03-25T17:58:57.959 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:57.959 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:57.959 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:57.959 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:57.959 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:57.959 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:57.959 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:57.959 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:57.959 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:57.959 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:57.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.958+0000 7f1c137fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:58:57.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.965+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 941413334 0 0) 0x7f1c18004eb0 con 0x7f1c2c0a98f0 2024-03-25T17:58:57.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.965+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c2c1351a0 con 0x7f1c2c0a98f0 2024-03-25T17:58:57.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.965+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f1c1c001a30 con 0x7f1c2c0a98f0 2024-03-25T17:58:57.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.965+0000 7f1c137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:57.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.965+0000 7f1c137fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:58:57.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.965+0000 7f1c137fe640 10 monclient(hunting): dump: 2024-03-25T17:58:57.967 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:57.967 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:57.967 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:57.967 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:57.968 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:57.968 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:57.968 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:57.968 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:57.968 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:57.968 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:57.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 602194864 0 0) 0x7f1c2c136e20 con 0x7f1c2c13f710 2024-03-25T17:58:57.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 >> v1:172.21.15.18:6790/0 conn(0x7f1c2c0a98f0 legacy=0x7f1c2c1daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:57.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 >> v1:172.21.15.18:6789/0 conn(0x7f1c2c1430c0 legacy=0x7f1c2c1e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:57.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c137fe640 10 monclient: _finish_hunting 0 2024-03-25T17:58:57.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c137fe640 1 monclient: found mon.b 2024-03-25T17:58:57.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:58:57.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c137fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:58:57.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c135330 con 0x7f1c2c13f710 2024-03-25T17:58:57.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c137fe640 10 monclient: _finish_auth 0 2024-03-25T17:58:57.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c137fe640 10 monclient: _check_auth_tickets 2024-03-25T17:58:57.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:58:57.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c31255640 5 monclient: authenticate success, global_id 4330 2024-03-25T17:58:57.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c31255640 10 monclient: _renew_subs 2024-03-25T17:58:57.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c31255640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:58:57.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c31255640 1 -- 172.21.15.18:0/1691791070 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2c135560 con 0x7f1c2c13f710 2024-03-25T17:58:57.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c31255640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:58:57.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c31255640 10 monclient: _renew_subs 2024-03-25T17:58:57.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c31255640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:58:57.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c31255640 1 -- 172.21.15.18:0/1691791070 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2c13b490 con 0x7f1c2c13f710 2024-03-25T17:58:57.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.973+0000 7f1c31255640 1 librados: init done 2024-03-25T17:58:57.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.974+0000 7f1c31255640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:57.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.974+0000 7f1c31255640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:57.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.974+0000 7f1c31255640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:58:57.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:57.974+0000 7f1c31255640 1 -- 172.21.15.18:0/1691791070 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2c0b7ad0 con 0x7f1c2c13f710 2024-03-25T17:58:58.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.263+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c20003610 con 0x7f1c2c13f710 2024-03-25T17:58:58.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.263+0000 7f1c137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:58:58.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.263+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f1c200039a0 con 0x7f1c2c13f710 2024-03-25T17:58:58.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.263+0000 7f1c137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:58:58.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.263+0000 7f1c137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:58:58.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.263+0000 7f1c137fe640 10 monclient: dump: 2024-03-25T17:58:58.265 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:58:58.265 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:58:58.265 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:58.265 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:58:58.265 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:58:58.265 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:58:58.266 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:58:58.266 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:58:58.266 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:58:58.266 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:58:58.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.263+0000 7f1c2959a640 4 set_mon_vals no callback set 2024-03-25T17:58:58.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.264+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f1c20003bc0 con 0x7f1c2c13f710 2024-03-25T17:58:58.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.265+0000 7f1c2ad9d640 10 monclient: get_auth_request con 0x7f1c1806df70 auth_method 0 2024-03-25T17:58:58.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.265+0000 7f1c137fe640 10 client.4330.objecter ms_handle_connect 0x7f1c1806df70 2024-03-25T17:58:58.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.269+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f1c20053540 con 0x7f1c2c13f710 2024-03-25T17:58:58.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.269+0000 7f1c137fe640 10 client.4330.objecter ms_dispatch 0x7f1c2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:58:58.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.269+0000 7f1c137fe640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:58:58.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.269+0000 7f1c137fe640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:58:58.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.269+0000 7f1c137fe640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-25T17:58:58.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.270+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c2004f050 con 0x7f1c2c13f710 2024-03-25T17:58:58.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.270+0000 7f1c137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:58:58.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.270+0000 7f1c137fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:58:58.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.522+0000 7f1c31255640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:58.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.522+0000 7f1c31255640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:58.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.522+0000 7f1c31255640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:58:58.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.522+0000 7f1c31255640 1 -- 172.21.15.18:0/1691791070 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c2c0b7730 con 0x7f1c2c13f710 2024-03-25T17:58:58.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.551+0000 7f1c137fe640 1 -- 172.21.15.18:0/1691791070 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f1c20056ba0 con 0x7f1c2c13f710 2024-03-25T17:58:58.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.551+0000 7f1c137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:58:58.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.551+0000 7f1c137fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:58:58.553 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:58:58.553 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:58:58.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.554+0000 7f1c31255640 10 librados: watch_flush enter 2024-03-25T17:58:58.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.554+0000 7f1c31255640 10 librados: watch_flush exit 2024-03-25T17:58:58.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.554+0000 7f1c31255640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-25T17:58:58.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.554+0000 7f1c31255640 10 client.4330.objecter successfully canceled tick 2024-03-25T17:58:58.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.554+0000 7f1c31255640 1 -- 172.21.15.18:0/1691791070 >> v1:172.21.15.18:6800/3721244451 conn(0x7f1c1806df70 legacy=0x7f1c18070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:58.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.554+0000 7f1c31255640 10 monclient: shutdown 2024-03-25T17:58:58.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.554+0000 7f1c31255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:58:58.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.554+0000 7f1c31255640 1 -- 172.21.15.18:0/1691791070 >> v1:172.21.15.169:6789/0 conn(0x7f1c2c13f710 legacy=0x7f1c2c1db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:58:58.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.554+0000 7f1c31255640 1 -- 172.21.15.18:0/1691791070 shutdown_connections 2024-03-25T17:58:58.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.554+0000 7f1c31255640 1 -- 172.21.15.18:0/1691791070 >> 172.21.15.18:0/1691791070 conn(0x7f1c2c0b7ea0 msgr2=0x7f1c2c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:58:58.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.554+0000 7f1c31255640 1 -- 172.21.15.18:0/1691791070 shutdown_connections 2024-03-25T17:58:58.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.554+0000 7f1c31255640 1 -- 172.21.15.18:0/1691791070 wait complete. 2024-03-25T17:58:58.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:58:58.554+0000 7f1c31255640 1 librados: shutdown 2024-03-25T17:58:58.569 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:59:02.436 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:00.230 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:02.500 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:00.293 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:02.533 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:00.326 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:02.559 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:03:00.309 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:02.560 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:00.353 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:02.562 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:03:00.311 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:02.572 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:00.365 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:02.589 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:03:00.338 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:02.604 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:00.397 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:05.377 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:59:05.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.493+0000 7f3a6f104640 10 monclient: get_monmap_and_config 2024-03-25T17:59:05.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.493+0000 7f3a6f104640 10 monclient: build_initial_monmap 2024-03-25T17:59:05.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.493+0000 7f3a6f104640 10 monclient: monmap: 2024-03-25T17:59:05.495 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:59:05.495 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:05.495 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:59:05.495324+0000 2024-03-25T17:59:05.495 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:59:05.495324+0000 2024-03-25T17:59:05.495 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:59:05.495 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:05.495 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:59:05.495 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:59:05.495 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:59:05.495 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:05.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.494+0000 7f3a6f104640 1 Processor -- start 2024-03-25T17:59:05.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.495+0000 7f3a6f104640 1 -- start start 2024-03-25T17:59:05.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.495+0000 7f3a6f104640 10 monclient: init 2024-03-25T17:59:05.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.495+0000 7f3a6f104640 10 monclient: _reopen_session rank -1 2024-03-25T17:59:05.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.495+0000 7f3a6f104640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:59:05.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.495+0000 7f3a6f104640 10 monclient(hunting): picked mon.noname-a con 0x7f3a680a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T17:59:05.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.495+0000 7f3a6f104640 10 monclient(hunting): picked mon.noname-c con 0x7f3a6813f710 addr v1:172.21.15.169:6789/0 2024-03-25T17:59:05.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.496+0000 7f3a6f104640 10 monclient(hunting): picked mon.noname-b con 0x7f3a681430c0 addr v1:172.21.15.18:6790/0 2024-03-25T17:59:05.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.496+0000 7f3a6f104640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a68142520 con 0x7f3a680a98f0 2024-03-25T17:59:05.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.496+0000 7f3a6f104640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a680b28a0 con 0x7f3a6813f710 2024-03-25T17:59:05.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.496+0000 7f3a6f104640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a680b2a30 con 0x7f3a681430c0 2024-03-25T17:59:05.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.496+0000 7f3a6f104640 10 monclient(hunting): _renew_subs 2024-03-25T17:59:05.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.496+0000 7f3a6f104640 10 monclient(hunting): authenticate will time out at 7074.074827s 2024-03-25T17:59:05.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.519+0000 7f3a6d6a1640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f3a681430c0 0x7f3a6814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:58938/0 (socket says 172.21.15.18:58938) 2024-03-25T17:59:05.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.519+0000 7f3a6d6a1640 1 -- 172.21.15.18:0/20506938 learned_addr learned my addr 172.21.15.18:0/20506938 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:59:05.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.537+0000 7f3a677fe640 1 -- 172.21.15.18:0/20506938 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951065450 0 0) 0x7f3a680b2a30 con 0x7f3a681430c0 2024-03-25T17:59:05.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.538+0000 7f3a677fe640 10 monclient(hunting): _negotiate 2024-03-25T17:59:05.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.538+0000 7f3a677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:05.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.538+0000 7f3a677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:05.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.538+0000 7f3a677fe640 10 monclient(hunting): my global_id is 4352 2024-03-25T17:59:05.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.538+0000 7f3a677fe640 1 -- 172.21.15.18:0/20506938 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a58003520 con 0x7f3a681430c0 2024-03-25T17:59:05.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.539+0000 7f3a677fe640 1 -- 172.21.15.18:0/20506938 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869622029 0 0) 0x7f3a68142520 con 0x7f3a680a98f0 2024-03-25T17:59:05.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.539+0000 7f3a677fe640 10 monclient(hunting): _negotiate 2024-03-25T17:59:05.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.539+0000 7f3a677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:05.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.539+0000 7f3a677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:05.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.539+0000 7f3a677fe640 10 monclient(hunting): my global_id is 4344 2024-03-25T17:59:05.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.539+0000 7f3a677fe640 1 -- 172.21.15.18:0/20506938 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a680b2a30 con 0x7f3a680a98f0 2024-03-25T17:59:05.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.543+0000 7f3a677fe640 1 -- 172.21.15.18:0/20506938 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136769753 0 0) 0x7f3a680b28a0 con 0x7f3a6813f710 2024-03-25T17:59:05.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.543+0000 7f3a677fe640 10 monclient(hunting): _negotiate 2024-03-25T17:59:05.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.543+0000 7f3a677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:05.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.543+0000 7f3a677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:05.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.543+0000 7f3a677fe640 10 monclient(hunting): my global_id is 4333 2024-03-25T17:59:05.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.543+0000 7f3a677fe640 1 -- 172.21.15.18:0/20506938 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a68142520 con 0x7f3a6813f710 2024-03-25T17:59:05.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.548+0000 7f3a677fe640 1 -- 172.21.15.18:0/20506938 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2972287493 0 0) 0x7f3a680b2a30 con 0x7f3a680a98f0 2024-03-25T17:59:05.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.548+0000 7f3a677fe640 1 -- 172.21.15.18:0/20506938 >> v1:172.21.15.18:6790/0 conn(0x7f3a681430c0 legacy=0x7f3a6814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:05.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.548+0000 7f3a677fe640 1 -- 172.21.15.18:0/20506938 >> v1:172.21.15.169:6789/0 conn(0x7f3a6813f710 legacy=0x7f3a681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:05.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.548+0000 7f3a677fe640 10 monclient: _finish_hunting 0 2024-03-25T17:59:05.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.548+0000 7f3a677fe640 1 monclient: found mon.noname-a 2024-03-25T17:59:05.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.548+0000 7f3a677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:59:05.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.548+0000 7f3a677fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T17:59:05.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.548+0000 7f3a677fe640 1 -- 172.21.15.18:0/20506938 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a680b44c0 con 0x7f3a680a98f0 2024-03-25T17:59:05.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.548+0000 7f3a677fe640 10 monclient: _finish_auth 0 2024-03-25T17:59:05.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.548+0000 7f3a677fe640 10 monclient: _check_auth_tickets 2024-03-25T17:59:05.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.548+0000 7f3a677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:59:05.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.548+0000 7f3a677fe640 1 -- 172.21.15.18:0/20506938 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3a5c001670 con 0x7f3a680a98f0 2024-03-25T17:59:05.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.548+0000 7f3a677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:05.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.548+0000 7f3a677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:59:05.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.548+0000 7f3a677fe640 10 monclient: dump: 2024-03-25T17:59:05.551 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:05.551 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:05.551 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:05.551 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:05.551 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:05.551 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:05.551 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:05.551 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:05.551 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:05.551 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:05.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.548+0000 7f3a6f104640 5 monclient: authenticate success, global_id 4344 2024-03-25T17:59:05.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.548+0000 7f3a6f104640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:59:05.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.558+0000 7f3a677fe640 1 -- 172.21.15.18:0/20506938 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a5c0017d0 con 0x7f3a680a98f0 2024-03-25T17:59:05.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.558+0000 7f3a677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:59:05.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.558+0000 7f3a677fe640 1 -- 172.21.15.18:0/20506938 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3a5c001be0 con 0x7f3a680a98f0 2024-03-25T17:59:05.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.558+0000 7f3a677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:05.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.558+0000 7f3a677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:59:05.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.558+0000 7f3a677fe640 10 monclient: dump: 2024-03-25T17:59:05.559 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:05.559 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:05.559 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:05.559 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:05.559 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:05.559 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:05.559 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:05.560 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:05.560 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:05.560 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:05.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.558+0000 7f3a6f104640 10 monclient: get_monmap_and_config success 2024-03-25T17:59:05.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.558+0000 7f3a6f104640 4 set_mon_vals no callback set 2024-03-25T17:59:05.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.558+0000 7f3a6f104640 10 monclient: shutdown 2024-03-25T17:59:05.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.558+0000 7f3a6f104640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:59:05.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.558+0000 7f3a6f104640 1 -- 172.21.15.18:0/20506938 >> v1:172.21.15.18:6789/0 conn(0x7f3a680a98f0 legacy=0x7f3a680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:05.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.558+0000 7f3a6f104640 1 -- 172.21.15.18:0/20506938 shutdown_connections 2024-03-25T17:59:05.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.558+0000 7f3a6f104640 1 -- 172.21.15.18:0/20506938 >> 172.21.15.18:0/20506938 conn(0x7f3a680b7ea0 msgr2=0x7f3a680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:59:05.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.559+0000 7f3a6f104640 1 -- 172.21.15.18:0/20506938 shutdown_connections 2024-03-25T17:59:05.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.559+0000 7f3a6f104640 1 -- 172.21.15.18:0/20506938 wait complete. 2024-03-25T17:59:05.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.559+0000 7f3a6f104640 10 monclient: build_initial_monmap 2024-03-25T17:59:05.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.559+0000 7f3a6f104640 10 monclient: monmap: 2024-03-25T17:59:05.563 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:59:05.563 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:59:05.563 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:59:05.563 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:59:05.563 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:59:05.563 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:59:05.563 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:59:05.564 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:59:05.564 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:59:05.564 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:05.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.560+0000 7f3a6f104640 1 librados: starting msgr at 2024-03-25T17:59:05.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.560+0000 7f3a6f104640 1 librados: starting objecter 2024-03-25T17:59:05.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.560+0000 7f3a6f104640 1 Processor -- start 2024-03-25T17:59:05.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.560+0000 7f3a6f104640 1 -- start start 2024-03-25T17:59:05.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.560+0000 7f3a6f104640 1 librados: setting wanted keys 2024-03-25T17:59:05.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.560+0000 7f3a6f104640 1 librados: calling monclient init 2024-03-25T17:59:05.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.560+0000 7f3a6f104640 10 monclient: init 2024-03-25T17:59:05.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.561+0000 7f3a6f104640 10 monclient: _reopen_session rank -1 2024-03-25T17:59:05.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.561+0000 7f3a6f104640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:59:05.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.561+0000 7f3a6f104640 10 monclient(hunting): picked mon.noname-b con 0x7f3a680a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T17:59:05.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.561+0000 7f3a6f104640 10 monclient(hunting): picked mon.noname-a con 0x7f3a6813f710 addr v1:172.21.15.18:6789/0 2024-03-25T17:59:05.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.561+0000 7f3a6f104640 10 monclient(hunting): picked mon.noname-c con 0x7f3a681430c0 addr v1:172.21.15.18:6790/0 2024-03-25T17:59:05.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.561+0000 7f3a6f104640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a68138f10 con 0x7f3a6813f710 2024-03-25T17:59:05.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.561+0000 7f3a6f104640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a68139970 con 0x7f3a680a98f0 2024-03-25T17:59:05.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.561+0000 7f3a6f104640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a680abb50 con 0x7f3a681430c0 2024-03-25T17:59:05.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.561+0000 7f3a6f104640 10 monclient(hunting): _renew_subs 2024-03-25T17:59:05.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.561+0000 7f3a6f104640 10 monclient(hunting): authenticate will time out at 7074.140222s 2024-03-25T17:59:05.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.569+0000 7f3a6cea0640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f3a680a98f0 0x7f3a68136330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:36156/0 (socket says 172.21.15.18:36156) 2024-03-25T17:59:05.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.570+0000 7f3a6cea0640 1 -- 172.21.15.18:0/1587244353 learned_addr learned my addr 172.21.15.18:0/1587244353 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:59:05.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.571+0000 7f3a657fa640 10 client.?.objecter ms_handle_connect 0x7f3a680a98f0 2024-03-25T17:59:05.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.571+0000 7f3a657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:05.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.579+0000 7f3a657fa640 10 client.?.objecter ms_handle_connect 0x7f3a6813f710 2024-03-25T17:59:05.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.579+0000 7f3a657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:05.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.583+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948036957 0 0) 0x7f3a68139970 con 0x7f3a680a98f0 2024-03-25T17:59:05.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.583+0000 7f3a657fa640 10 monclient(hunting): _negotiate 2024-03-25T17:59:05.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.583+0000 7f3a657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:05.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.583+0000 7f3a657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:05.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.583+0000 7f3a657fa640 10 monclient(hunting): my global_id is 4336 2024-03-25T17:59:05.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.583+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a4c004eb0 con 0x7f3a680a98f0 2024-03-25T17:59:05.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.584+0000 7f3a657fa640 10 client.?.objecter ms_handle_connect 0x7f3a681430c0 2024-03-25T17:59:05.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.584+0000 7f3a657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:05.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.593+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354758682 0 0) 0x7f3a68138f10 con 0x7f3a6813f710 2024-03-25T17:59:05.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.593+0000 7f3a657fa640 10 monclient(hunting): _negotiate 2024-03-25T17:59:05.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.593+0000 7f3a657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:05.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.593+0000 7f3a657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:05.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.593+0000 7f3a657fa640 10 monclient(hunting): my global_id is 4347 2024-03-25T17:59:05.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.593+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a68139970 con 0x7f3a6813f710 2024-03-25T17:59:05.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.604+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2272133320 0 0) 0x7f3a4c004eb0 con 0x7f3a680a98f0 2024-03-25T17:59:05.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.604+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a68138f10 con 0x7f3a680a98f0 2024-03-25T17:59:05.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.604+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3a5c001670 con 0x7f3a680a98f0 2024-03-25T17:59:05.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.604+0000 7f3a657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:05.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.604+0000 7f3a657fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:59:05.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.604+0000 7f3a657fa640 10 monclient(hunting): dump: 2024-03-25T17:59:05.605 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:05.605 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:05.605 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:05.606 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:05.606 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:05.606 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:05.606 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:05.606 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:05.606 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:05.606 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:05.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.604+0000 7f3a657fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:59:05.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.605+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518410032 0 0) 0x7f3a680abb50 con 0x7f3a681430c0 2024-03-25T17:59:05.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.605+0000 7f3a657fa640 10 monclient(hunting): _negotiate 2024-03-25T17:59:05.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.605+0000 7f3a657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:05.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.605+0000 7f3a657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:05.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.605+0000 7f3a657fa640 10 monclient(hunting): my global_id is 4355 2024-03-25T17:59:05.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.605+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a4c004eb0 con 0x7f3a681430c0 2024-03-25T17:59:05.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.609+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1220779474 0 0) 0x7f3a68139970 con 0x7f3a6813f710 2024-03-25T17:59:05.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.609+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a680abb50 con 0x7f3a6813f710 2024-03-25T17:59:05.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.609+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3a54001870 con 0x7f3a6813f710 2024-03-25T17:59:05.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.609+0000 7f3a657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:05.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.609+0000 7f3a657fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:59:05.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.609+0000 7f3a657fa640 10 monclient(hunting): dump: 2024-03-25T17:59:05.610 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:05.610 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:05.610 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:05.610 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:05.610 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:05.610 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:05.611 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:05.611 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:05.611 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:05.611 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:05.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.611+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 667437848 0 0) 0x7f3a68138f10 con 0x7f3a680a98f0 2024-03-25T17:59:05.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.611+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 >> v1:172.21.15.18:6790/0 conn(0x7f3a681430c0 legacy=0x7f3a68138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:05.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.611+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 >> v1:172.21.15.18:6789/0 conn(0x7f3a6813f710 legacy=0x7f3a680ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:05.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.611+0000 7f3a657fa640 10 monclient: _finish_hunting 0 2024-03-25T17:59:05.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.611+0000 7f3a657fa640 1 monclient: found mon.b 2024-03-25T17:59:05.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.611+0000 7f3a657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:59:05.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.611+0000 7f3a657fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:05.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.611+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a680abe30 con 0x7f3a680a98f0 2024-03-25T17:59:05.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.611+0000 7f3a657fa640 10 monclient: _finish_auth 0 2024-03-25T17:59:05.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.611+0000 7f3a657fa640 10 monclient: _check_auth_tickets 2024-03-25T17:59:05.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.611+0000 7f3a657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:59:05.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.611+0000 7f3a6f104640 5 monclient: authenticate success, global_id 4336 2024-03-25T17:59:05.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.611+0000 7f3a6f104640 10 monclient: _renew_subs 2024-03-25T17:59:05.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.611+0000 7f3a6f104640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:05.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.611+0000 7f3a6f104640 1 -- 172.21.15.18:0/1587244353 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a68137b40 con 0x7f3a680a98f0 2024-03-25T17:59:05.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.612+0000 7f3a6f104640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:59:05.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.612+0000 7f3a6f104640 10 monclient: _renew_subs 2024-03-25T17:59:05.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.612+0000 7f3a6f104640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:05.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.612+0000 7f3a6f104640 1 -- 172.21.15.18:0/1587244353 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a68138c10 con 0x7f3a680a98f0 2024-03-25T17:59:05.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.612+0000 7f3a6f104640 1 librados: init done 2024-03-25T17:59:05.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.613+0000 7f3a6f104640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:05.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.613+0000 7f3a6f104640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:05.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.613+0000 7f3a6f104640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:05.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.613+0000 7f3a6f104640 1 -- 172.21.15.18:0/1587244353 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a680627b0 con 0x7f3a680a98f0 2024-03-25T17:59:05.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.912+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a5c0038a0 con 0x7f3a680a98f0 2024-03-25T17:59:05.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.912+0000 7f3a657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:59:05.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.912+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3a5c003cf0 con 0x7f3a680a98f0 2024-03-25T17:59:05.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.912+0000 7f3a657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:05.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.912+0000 7f3a657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:59:05.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.912+0000 7f3a657fa640 10 monclient: dump: 2024-03-25T17:59:05.913 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:05.913 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:05.913 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:05.913 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:05.913 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:05.913 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:05.913 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:05.914 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:05.914 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:05.914 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:05.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.912+0000 7f3a66ffd640 4 set_mon_vals no callback set 2024-03-25T17:59:05.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.913+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f3a5c003f10 con 0x7f3a680a98f0 2024-03-25T17:59:05.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.913+0000 7f3a67fff640 10 monclient: get_auth_request con 0x7f3a4c06dfb0 auth_method 0 2024-03-25T17:59:05.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.913+0000 7f3a657fa640 10 client.4336.objecter ms_handle_connect 0x7f3a4c06dfb0 2024-03-25T17:59:05.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.917+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f3a5c054960 con 0x7f3a680a98f0 2024-03-25T17:59:05.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.917+0000 7f3a657fa640 10 client.4336.objecter ms_dispatch 0x7f3a68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:59:05.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.917+0000 7f3a657fa640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:59:05.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.917+0000 7f3a657fa640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:59:05.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.917+0000 7f3a657fa640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T17:59:05.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.919+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3a5c050470 con 0x7f3a680a98f0 2024-03-25T17:59:05.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.919+0000 7f3a657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:05.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:05.919+0000 7f3a657fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:59:06.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.155+0000 7f3a6f104640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:06.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.155+0000 7f3a6f104640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:06.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.155+0000 7f3a6f104640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:06.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.155+0000 7f3a6f104640 1 -- 172.21.15.18:0/1587244353 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a6813db40 con 0x7f3a680a98f0 2024-03-25T17:59:06.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.200+0000 7f3a657fa640 1 -- 172.21.15.18:0/1587244353 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f3a5c0523a0 con 0x7f3a680a98f0 2024-03-25T17:59:06.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.200+0000 7f3a657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:06.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.200+0000 7f3a657fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:59:06.201 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:59:06.201 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:59:06.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.202+0000 7f3a6f104640 10 librados: watch_flush enter 2024-03-25T17:59:06.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.202+0000 7f3a6f104640 10 librados: watch_flush exit 2024-03-25T17:59:06.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.202+0000 7f3a6f104640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T17:59:06.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.202+0000 7f3a6f104640 10 client.4336.objecter successfully canceled tick 2024-03-25T17:59:06.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.202+0000 7f3a6f104640 1 -- 172.21.15.18:0/1587244353 >> v1:172.21.15.18:6800/3721244451 conn(0x7f3a4c06dfb0 legacy=0x7f3a4c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:06.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.202+0000 7f3a6f104640 10 monclient: shutdown 2024-03-25T17:59:06.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.202+0000 7f3a6f104640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:59:06.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.202+0000 7f3a6f104640 1 -- 172.21.15.18:0/1587244353 >> v1:172.21.15.169:6789/0 conn(0x7f3a680a98f0 legacy=0x7f3a68136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:06.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.202+0000 7f3a6f104640 1 -- 172.21.15.18:0/1587244353 shutdown_connections 2024-03-25T17:59:06.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.202+0000 7f3a6f104640 1 -- 172.21.15.18:0/1587244353 >> 172.21.15.18:0/1587244353 conn(0x7f3a680b7ea0 msgr2=0x7f3a68074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:59:06.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.202+0000 7f3a6f104640 1 -- 172.21.15.18:0/1587244353 shutdown_connections 2024-03-25T17:59:06.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.202+0000 7f3a6f104640 1 -- 172.21.15.18:0/1587244353 wait complete. 2024-03-25T17:59:06.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:06.202+0000 7f3a6f104640 1 librados: shutdown 2024-03-25T17:59:06.215 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:59:13.022 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:59:13.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.120+0000 7f98c47ba640 10 monclient: get_monmap_and_config 2024-03-25T17:59:13.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.120+0000 7f98c47ba640 10 monclient: build_initial_monmap 2024-03-25T17:59:13.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.120+0000 7f98c47ba640 10 monclient: monmap: 2024-03-25T17:59:13.121 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:59:13.121 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:13.121 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:59:13.121583+0000 2024-03-25T17:59:13.121 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:59:13.121583+0000 2024-03-25T17:59:13.121 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:59:13.121 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:13.122 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:59:13.122 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:59:13.122 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:59:13.122 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:13.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.121+0000 7f98c47ba640 1 Processor -- start 2024-03-25T17:59:13.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.121+0000 7f98c47ba640 1 -- start start 2024-03-25T17:59:13.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.121+0000 7f98c47ba640 10 monclient: init 2024-03-25T17:59:13.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.121+0000 7f98c47ba640 10 monclient: _reopen_session rank -1 2024-03-25T17:59:13.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.121+0000 7f98c47ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:59:13.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.121+0000 7f98c47ba640 10 monclient(hunting): picked mon.noname-a con 0x7f98bc0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T17:59:13.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.121+0000 7f98c47ba640 10 monclient(hunting): picked mon.noname-c con 0x7f98bc13f710 addr v1:172.21.15.169:6789/0 2024-03-25T17:59:13.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.122+0000 7f98c47ba640 10 monclient(hunting): picked mon.noname-b con 0x7f98bc1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T17:59:13.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.122+0000 7f98c47ba640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98bc142520 con 0x7f98bc0a98f0 2024-03-25T17:59:13.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.122+0000 7f98c47ba640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98bc0b28a0 con 0x7f98bc13f710 2024-03-25T17:59:13.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.122+0000 7f98c47ba640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98bc0b2a30 con 0x7f98bc1430c0 2024-03-25T17:59:13.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.122+0000 7f98c47ba640 10 monclient(hunting): _renew_subs 2024-03-25T17:59:13.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.122+0000 7f98c47ba640 10 monclient(hunting): authenticate will time out at 7081.700793s 2024-03-25T17:59:13.138 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.137+0000 7f98c2556640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f98bc0a98f0 0x7f98bc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:41934/0 (socket says 172.21.15.18:41934) 2024-03-25T17:59:13.138 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.137+0000 7f98c2556640 1 -- 172.21.15.18:0/1935336964 learned_addr learned my addr 172.21.15.18:0/1935336964 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:59:13.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.151+0000 7f98c1554640 1 -- 172.21.15.18:0/1935336964 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243201457 0 0) 0x7f98bc142520 con 0x7f98bc0a98f0 2024-03-25T17:59:13.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.151+0000 7f98c1554640 10 monclient(hunting): _negotiate 2024-03-25T17:59:13.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.151+0000 7f98c1554640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:13.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.151+0000 7f98c1554640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:13.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.151+0000 7f98c1554640 10 monclient(hunting): my global_id is 4350 2024-03-25T17:59:13.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.151+0000 7f98c1554640 1 -- 172.21.15.18:0/1935336964 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98a8003520 con 0x7f98bc0a98f0 2024-03-25T17:59:13.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c1554640 1 -- 172.21.15.18:0/1935336964 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 184965851 0 0) 0x7f98a8003520 con 0x7f98bc0a98f0 2024-03-25T17:59:13.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c1554640 1 -- 172.21.15.18:0/1935336964 >> v1:172.21.15.18:6790/0 conn(0x7f98bc1430c0 legacy=0x7f98bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:13.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c1554640 1 -- 172.21.15.18:0/1935336964 >> v1:172.21.15.169:6789/0 conn(0x7f98bc13f710 legacy=0x7f98bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:13.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c1554640 10 monclient: _finish_hunting 0 2024-03-25T17:59:13.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c1554640 1 monclient: found mon.noname-a 2024-03-25T17:59:13.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c1554640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:59:13.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c1554640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T17:59:13.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c1554640 1 -- 172.21.15.18:0/1935336964 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc0b44c0 con 0x7f98bc0a98f0 2024-03-25T17:59:13.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c1554640 10 monclient: _finish_auth 0 2024-03-25T17:59:13.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c1554640 10 monclient: _check_auth_tickets 2024-03-25T17:59:13.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c1554640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:59:13.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c1554640 1 -- 172.21.15.18:0/1935336964 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f98b8001690 con 0x7f98bc0a98f0 2024-03-25T17:59:13.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c47ba640 5 monclient: authenticate success, global_id 4350 2024-03-25T17:59:13.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c47ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:59:13.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c1554640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:13.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c1554640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:59:13.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c1554640 10 monclient: dump: 2024-03-25T17:59:13.169 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:13.169 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:13.169 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:13.169 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:13.169 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:13.169 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:13.169 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:13.169 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:13.169 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:13.169 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:13.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.166+0000 7f98c47ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:59:13.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.178+0000 7f98c1554640 1 -- 172.21.15.18:0/1935336964 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98b80017f0 con 0x7f98bc0a98f0 2024-03-25T17:59:13.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.178+0000 7f98c1554640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:59:13.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.178+0000 7f98c1554640 1 -- 172.21.15.18:0/1935336964 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f98b8001c00 con 0x7f98bc0a98f0 2024-03-25T17:59:13.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.178+0000 7f98c47ba640 10 monclient: get_monmap_and_config success 2024-03-25T17:59:13.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.178+0000 7f98c47ba640 4 set_mon_vals no callback set 2024-03-25T17:59:13.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.178+0000 7f98c1554640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:13.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.178+0000 7f98c1554640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:59:13.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.178+0000 7f98c1554640 10 monclient: dump: 2024-03-25T17:59:13.180 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:13.180 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:13.180 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:13.180 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:13.180 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:13.180 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:13.180 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:13.180 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:13.180 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:13.180 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:13.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.179+0000 7f98c47ba640 10 monclient: shutdown 2024-03-25T17:59:13.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.179+0000 7f98c47ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:59:13.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.179+0000 7f98c47ba640 1 -- 172.21.15.18:0/1935336964 >> v1:172.21.15.18:6789/0 conn(0x7f98bc0a98f0 legacy=0x7f98bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:13.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.179+0000 7f98c47ba640 1 -- 172.21.15.18:0/1935336964 shutdown_connections 2024-03-25T17:59:13.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.179+0000 7f98c47ba640 1 -- 172.21.15.18:0/1935336964 >> 172.21.15.18:0/1935336964 conn(0x7f98bc0b7ea0 msgr2=0x7f98bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:59:13.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.179+0000 7f98c47ba640 1 -- 172.21.15.18:0/1935336964 shutdown_connections 2024-03-25T17:59:13.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.179+0000 7f98c47ba640 1 -- 172.21.15.18:0/1935336964 wait complete. 2024-03-25T17:59:13.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.180+0000 7f98c47ba640 10 monclient: build_initial_monmap 2024-03-25T17:59:13.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.180+0000 7f98c47ba640 10 monclient: monmap: 2024-03-25T17:59:13.183 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:59:13.183 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:59:13.183 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:59:13.183 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:59:13.183 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:59:13.183 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:59:13.183 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:59:13.183 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:59:13.183 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:59:13.184 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:13.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.180+0000 7f98c47ba640 1 librados: starting msgr at 2024-03-25T17:59:13.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.180+0000 7f98c47ba640 1 librados: starting objecter 2024-03-25T17:59:13.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.180+0000 7f98c47ba640 1 Processor -- start 2024-03-25T17:59:13.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.180+0000 7f98c47ba640 1 -- start start 2024-03-25T17:59:13.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.180+0000 7f98c47ba640 1 librados: setting wanted keys 2024-03-25T17:59:13.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.180+0000 7f98c47ba640 1 librados: calling monclient init 2024-03-25T17:59:13.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.180+0000 7f98c47ba640 10 monclient: init 2024-03-25T17:59:13.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.181+0000 7f98c47ba640 10 monclient: _reopen_session rank -1 2024-03-25T17:59:13.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.181+0000 7f98c47ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:59:13.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.181+0000 7f98c47ba640 10 monclient(hunting): picked mon.noname-b con 0x7f98bc0a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T17:59:13.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.181+0000 7f98c47ba640 10 monclient(hunting): picked mon.noname-a con 0x7f98bc13f710 addr v1:172.21.15.18:6789/0 2024-03-25T17:59:13.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.181+0000 7f98c47ba640 10 monclient(hunting): picked mon.noname-c con 0x7f98bc1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T17:59:13.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.181+0000 7f98c47ba640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98bc0abd30 con 0x7f98bc13f710 2024-03-25T17:59:13.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.181+0000 7f98c47ba640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98bc1d77a0 con 0x7f98bc0a98f0 2024-03-25T17:59:13.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.181+0000 7f98c47ba640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98bc1d51f0 con 0x7f98bc1430c0 2024-03-25T17:59:13.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.181+0000 7f98c47ba640 10 monclient(hunting): _renew_subs 2024-03-25T17:59:13.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.181+0000 7f98c47ba640 10 monclient(hunting): authenticate will time out at 7081.760372s 2024-03-25T17:59:13.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.190+0000 7f98c2556640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f98bc0a98f0 0x7f98bc1d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:40892/0 (socket says 172.21.15.18:40892) 2024-03-25T17:59:13.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.190+0000 7f98c2556640 1 -- 172.21.15.18:0/3566332356 learned_addr learned my addr 172.21.15.18:0/3566332356 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:59:13.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.191+0000 7f98b2ffd640 10 client.?.objecter ms_handle_connect 0x7f98bc0a98f0 2024-03-25T17:59:13.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.191+0000 7f98b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:13.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.198+0000 7f98b2ffd640 10 client.?.objecter ms_handle_connect 0x7f98bc1430c0 2024-03-25T17:59:13.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.198+0000 7f98b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:13.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.200+0000 7f98b2ffd640 10 client.?.objecter ms_handle_connect 0x7f98bc13f710 2024-03-25T17:59:13.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.200+0000 7f98b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:13.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.200+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860607369 0 0) 0x7f98bc1d77a0 con 0x7f98bc0a98f0 2024-03-25T17:59:13.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.200+0000 7f98b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:59:13.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.200+0000 7f98b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:13.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.200+0000 7f98b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:13.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.200+0000 7f98b2ffd640 10 monclient(hunting): my global_id is 4342 2024-03-25T17:59:13.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.200+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98a0004eb0 con 0x7f98bc0a98f0 2024-03-25T17:59:13.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.209+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1348315667 0 0) 0x7f98a0004eb0 con 0x7f98bc0a98f0 2024-03-25T17:59:13.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.209+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98bc1d77a0 con 0x7f98bc0a98f0 2024-03-25T17:59:13.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.209+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f98b8001bc0 con 0x7f98bc0a98f0 2024-03-25T17:59:13.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.209+0000 7f98b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:13.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.209+0000 7f98b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:59:13.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.209+0000 7f98b2ffd640 10 monclient(hunting): dump: 2024-03-25T17:59:13.210 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:13.210 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:13.210 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:13.210 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:13.211 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:13.211 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:13.211 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:13.211 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:13.211 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:13.211 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:13.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.209+0000 7f98b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:59:13.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.211+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276173964 0 0) 0x7f98bc1d51f0 con 0x7f98bc1430c0 2024-03-25T17:59:13.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.211+0000 7f98b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:59:13.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.211+0000 7f98b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:13.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.211+0000 7f98b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:13.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.211+0000 7f98b2ffd640 10 monclient(hunting): my global_id is 4361 2024-03-25T17:59:13.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.211+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98a0004eb0 con 0x7f98bc1430c0 2024-03-25T17:59:13.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.212+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355190483 0 0) 0x7f98bc0abd30 con 0x7f98bc13f710 2024-03-25T17:59:13.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.212+0000 7f98b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:59:13.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.212+0000 7f98b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:13.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.212+0000 7f98b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:13.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.212+0000 7f98b2ffd640 10 monclient(hunting): my global_id is 4353 2024-03-25T17:59:13.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.212+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98bc1d51f0 con 0x7f98bc13f710 2024-03-25T17:59:13.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.232+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2521818759 0 0) 0x7f98bc1d77a0 con 0x7f98bc0a98f0 2024-03-25T17:59:13.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.232+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 >> v1:172.21.15.18:6790/0 conn(0x7f98bc1430c0 legacy=0x7f98bc1e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:13.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.232+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 >> v1:172.21.15.18:6789/0 conn(0x7f98bc13f710 legacy=0x7f98bc138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:13.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.232+0000 7f98b2ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:59:13.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.232+0000 7f98b2ffd640 1 monclient: found mon.b 2024-03-25T17:59:13.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.232+0000 7f98b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:59:13.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.232+0000 7f98b2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:13.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.232+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc1d83d0 con 0x7f98bc0a98f0 2024-03-25T17:59:13.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.232+0000 7f98b2ffd640 10 monclient: _finish_auth 0 2024-03-25T17:59:13.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.232+0000 7f98b2ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:59:13.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.232+0000 7f98b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:59:13.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.233+0000 7f98c47ba640 5 monclient: authenticate success, global_id 4342 2024-03-25T17:59:13.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.233+0000 7f98c47ba640 10 monclient: _renew_subs 2024-03-25T17:59:13.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.233+0000 7f98c47ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:13.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.233+0000 7f98c47ba640 1 -- 172.21.15.18:0/3566332356 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98bc138a50 con 0x7f98bc0a98f0 2024-03-25T17:59:13.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.233+0000 7f98c47ba640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:59:13.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.233+0000 7f98c47ba640 10 monclient: _renew_subs 2024-03-25T17:59:13.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.233+0000 7f98c47ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:13.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.233+0000 7f98c47ba640 1 -- 172.21.15.18:0/3566332356 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98bc1e35c0 con 0x7f98bc0a98f0 2024-03-25T17:59:13.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.233+0000 7f98c47ba640 1 librados: init done 2024-03-25T17:59:13.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.234+0000 7f98c47ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:13.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.234+0000 7f98c47ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:13.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.234+0000 7f98c47ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:13.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.234+0000 7f98c47ba640 1 -- 172.21.15.18:0/3566332356 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98bc0627b0 con 0x7f98bc0a98f0 2024-03-25T17:59:13.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.236+0000 7f98c2556640 1 -- 172.21.15.18:0/3566332356 >> v1:172.21.15.169:6789/0 conn(0x7f98bc0a98f0 legacy=0x7f98bc1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T17:59:13.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.236+0000 7f98c2556640 1 -- 172.21.15.18:0/3566332356 >> v1:172.21.15.169:6789/0 conn(0x7f98bc0a98f0 legacy=0x7f98bc1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T17:59:13.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.236+0000 7f98c2556640 1 --1- 172.21.15.18:0/3566332356 >> v1:172.21.15.169:6789/0 conn(0x7f98bc0a98f0 0x7f98bc1d7cc0 :-1 s=OPENED pgs=16297 cs=1 l=1).handle_message read tag failed 2024-03-25T17:59:13.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.236+0000 7f98c2556640 1 --1- 172.21.15.18:0/3566332356 >> v1:172.21.15.169:6789/0 conn(0x7f98bc0a98f0 0x7f98bc1d7cc0 :-1 s=OPENED pgs=16297 cs=1 l=1).fault on lossy channel, failing 2024-03-25T17:59:13.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.236+0000 7f98b2ffd640 10 monclient: ms_handle_reset current mon v1:172.21.15.169:6789/0 2024-03-25T17:59:13.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.236+0000 7f98b2ffd640 10 monclient: _reopen_session rank -1 2024-03-25T17:59:13.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.236+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 >> v1:172.21.15.169:6789/0 conn(0x7f98bc0a98f0 legacy=0x7f98bc1d7cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T17:59:13.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.236+0000 7f98b2ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:59:13.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.236+0000 7f98b2ffd640 10 monclient(hunting): picked mon.c con 0x7f98bc1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T17:59:13.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.236+0000 7f98b2ffd640 10 monclient(hunting): picked mon.a con 0x7f98bc13f710 addr v1:172.21.15.18:6789/0 2024-03-25T17:59:13.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.236+0000 7f98b2ffd640 10 monclient(hunting): picked mon.b con 0x7f98a000da30 addr v1:172.21.15.169:6789/0 2024-03-25T17:59:13.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.236+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f98bc1d77a0 con 0x7f98bc13f710 2024-03-25T17:59:13.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.236+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f98bc0abd30 con 0x7f98a000da30 2024-03-25T17:59:13.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.236+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f98a0013a70 con 0x7f98bc1430c0 2024-03-25T17:59:13.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.236+0000 7f98b2ffd640 10 monclient(hunting): _renew_subs 2024-03-25T17:59:13.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.246+0000 7f98b2ffd640 10 client.4342.objecter ms_handle_connect 0x7f98bc13f710 2024-03-25T17:59:13.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.246+0000 7f98b2ffd640 10 client.4342.objecter resend_mon_ops 2024-03-25T17:59:13.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.258+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629867368 0 0) 0x7f98bc1d77a0 con 0x7f98bc13f710 2024-03-25T17:59:13.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.258+0000 7f98b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:59:13.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.258+0000 7f98b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:13.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.258+0000 7f98b2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T17:59:13.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.258+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 --> v1:172.21.15.18:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f98a00119c0 con 0x7f98bc13f710 2024-03-25T17:59:13.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.264+0000 7f98b2ffd640 10 client.4342.objecter ms_handle_connect 0x7f98bc1430c0 2024-03-25T17:59:13.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.265+0000 7f98b2ffd640 10 client.4342.objecter resend_mon_ops 2024-03-25T17:59:13.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1717618762 0 0) 0x7f98a00119c0 con 0x7f98bc13f710 2024-03-25T17:59:13.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 >> v1:172.21.15.18:6790/0 conn(0x7f98bc1430c0 legacy=0x7f98a0006440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:13.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 >> v1:172.21.15.169:6789/0 conn(0x7f98a000da30 legacy=0x7f98a000fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:13.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:59:13.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 1 monclient: found mon.a 2024-03-25T17:59:13.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:59:13.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:13.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f98a0014e50 con 0x7f98bc13f710 2024-03-25T17:59:13.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:13.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:13.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98bc0abd30 con 0x7f98bc13f710 2024-03-25T17:59:13.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 10 monclient: _finish_auth 0 2024-03-25T17:59:13.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:59:13.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:59:13.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f98ac0016f0 con 0x7f98bc13f710 2024-03-25T17:59:13.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:13.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:59:13.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.268+0000 7f98b2ffd640 10 monclient: dump: 2024-03-25T17:59:13.271 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:13.271 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:13.271 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:13.271 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:13.271 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:13.271 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:13.271 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:13.272 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:13.272 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:13.272 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:13.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.594+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98ac001850 con 0x7f98bc13f710 2024-03-25T17:59:13.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.594+0000 7f98b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:59:13.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.594+0000 7f98c0d53640 4 set_mon_vals no callback set 2024-03-25T17:59:13.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.595+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f98ac001b00 con 0x7f98bc13f710 2024-03-25T17:59:13.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.596+0000 7f98c2556640 10 monclient: get_auth_request con 0x7f98bc0a98f0 auth_method 0 2024-03-25T17:59:13.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.596+0000 7f98b2ffd640 10 client.4342.objecter ms_handle_connect 0x7f98bc0a98f0 2024-03-25T17:59:13.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.602+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 <== mon.0 v1:172.21.15.18:6789/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f98ac052e50 con 0x7f98bc13f710 2024-03-25T17:59:13.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.602+0000 7f98b2ffd640 10 client.4342.objecter ms_dispatch 0x7f98bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:59:13.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.602+0000 7f98b2ffd640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:59:13.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.602+0000 7f98b2ffd640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:59:13.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.602+0000 7f98b2ffd640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-25T17:59:13.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.602+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98ac04e790 con 0x7f98bc13f710 2024-03-25T17:59:13.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.602+0000 7f98b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:13.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.602+0000 7f98b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:59:13.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.863+0000 7f98c47ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:13.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.863+0000 7f98c47ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:13.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.863+0000 7f98c47ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:13.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.863+0000 7f98c47ba640 1 -- 172.21.15.18:0/3566332356 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98bc13e250 con 0x7f98bc13f710 2024-03-25T17:59:13.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.890+0000 7f98b2ffd640 1 -- 172.21.15.18:0/3566332356 <== mon.0 v1:172.21.15.18:6789/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f98ac050890 con 0x7f98bc13f710 2024-03-25T17:59:13.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.890+0000 7f98b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:13.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.890+0000 7f98b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:59:13.891 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:59:13.891 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:59:13.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.892+0000 7f98c47ba640 10 librados: watch_flush enter 2024-03-25T17:59:13.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.892+0000 7f98c47ba640 10 librados: watch_flush exit 2024-03-25T17:59:13.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.892+0000 7f98c47ba640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-25T17:59:13.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.892+0000 7f98c47ba640 10 client.4342.objecter successfully canceled tick 2024-03-25T17:59:13.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.892+0000 7f98c47ba640 1 -- 172.21.15.18:0/3566332356 >> v1:172.21.15.18:6800/3721244451 conn(0x7f98bc0a98f0 legacy=0x7f98a007bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:13.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.892+0000 7f98c47ba640 10 monclient: shutdown 2024-03-25T17:59:13.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.892+0000 7f98c47ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:59:13.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.892+0000 7f98c47ba640 1 -- 172.21.15.18:0/3566332356 >> v1:172.21.15.18:6789/0 conn(0x7f98bc13f710 legacy=0x7f98a000c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:13.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.892+0000 7f98c47ba640 1 -- 172.21.15.18:0/3566332356 shutdown_connections 2024-03-25T17:59:13.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.892+0000 7f98c47ba640 1 -- 172.21.15.18:0/3566332356 >> 172.21.15.18:0/3566332356 conn(0x7f98bc0b7ea0 msgr2=0x7f98bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:59:13.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.893+0000 7f98c47ba640 1 -- 172.21.15.18:0/3566332356 shutdown_connections 2024-03-25T17:59:13.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.893+0000 7f98c47ba640 1 -- 172.21.15.18:0/3566332356 wait complete. 2024-03-25T17:59:13.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:13.893+0000 7f98c47ba640 1 librados: shutdown 2024-03-25T17:59:13.905 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:59:20.713 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:59:20.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.831+0000 7f366cbef640 10 monclient: get_monmap_and_config 2024-03-25T17:59:20.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.831+0000 7f366cbef640 10 monclient: build_initial_monmap 2024-03-25T17:59:20.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.831+0000 7f366cbef640 10 monclient: monmap: 2024-03-25T17:59:20.832 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:59:20.832 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:20.832 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:59:20.832609+0000 2024-03-25T17:59:20.832 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:59:20.832609+0000 2024-03-25T17:59:20.833 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:59:20.833 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:20.833 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:59:20.833 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:59:20.833 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:59:20.833 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:20.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.832+0000 7f366cbef640 1 Processor -- start 2024-03-25T17:59:20.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.832+0000 7f366cbef640 1 -- start start 2024-03-25T17:59:20.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.832+0000 7f366cbef640 10 monclient: init 2024-03-25T17:59:20.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.832+0000 7f366cbef640 10 monclient: _reopen_session rank -1 2024-03-25T17:59:20.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.832+0000 7f366cbef640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:59:20.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.833+0000 7f366cbef640 10 monclient(hunting): picked mon.noname-b con 0x7f36680a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T17:59:20.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.833+0000 7f366cbef640 10 monclient(hunting): picked mon.noname-a con 0x7f366813f710 addr v1:172.21.15.18:6789/0 2024-03-25T17:59:20.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.833+0000 7f366cbef640 10 monclient(hunting): picked mon.noname-c con 0x7f36681430c0 addr v1:172.21.15.169:6789/0 2024-03-25T17:59:20.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.833+0000 7f366cbef640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3668142520 con 0x7f366813f710 2024-03-25T17:59:20.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.833+0000 7f366cbef640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36680b28a0 con 0x7f36681430c0 2024-03-25T17:59:20.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.833+0000 7f366cbef640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36680b2a30 con 0x7f36680a98f0 2024-03-25T17:59:20.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.833+0000 7f366cbef640 10 monclient(hunting): _renew_subs 2024-03-25T17:59:20.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.833+0000 7f366cbef640 10 monclient(hunting): authenticate will time out at 7089.411934s 2024-03-25T17:59:20.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.840+0000 7f3666d9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f36681430c0 0x7f366814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:42198/0 (socket says 172.21.15.18:42198) 2024-03-25T17:59:20.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.840+0000 7f3666d9d640 1 -- 172.21.15.18:0/3329395044 learned_addr learned my addr 172.21.15.18:0/3329395044 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:59:20.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.849+0000 7f366559a640 1 -- 172.21.15.18:0/3329395044 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955162326 0 0) 0x7f36680b28a0 con 0x7f36681430c0 2024-03-25T17:59:20.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.849+0000 7f366559a640 10 monclient(hunting): _negotiate 2024-03-25T17:59:20.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.849+0000 7f366559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:20.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.849+0000 7f366559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:20.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.849+0000 7f366559a640 10 monclient(hunting): my global_id is 4345 2024-03-25T17:59:20.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.849+0000 7f366559a640 1 -- 172.21.15.18:0/3329395044 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3650003520 con 0x7f36681430c0 2024-03-25T17:59:20.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.859+0000 7f366559a640 1 -- 172.21.15.18:0/3329395044 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887793622 0 0) 0x7f3668142520 con 0x7f366813f710 2024-03-25T17:59:20.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.859+0000 7f366559a640 10 monclient(hunting): _negotiate 2024-03-25T17:59:20.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.859+0000 7f366559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:20.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.859+0000 7f366559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:20.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.859+0000 7f366559a640 10 monclient(hunting): my global_id is 4356 2024-03-25T17:59:20.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.859+0000 7f366559a640 1 -- 172.21.15.18:0/3329395044 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36680b28a0 con 0x7f366813f710 2024-03-25T17:59:20.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.860+0000 7f366559a640 1 -- 172.21.15.18:0/3329395044 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156863417 0 0) 0x7f36680b2a30 con 0x7f36680a98f0 2024-03-25T17:59:20.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.860+0000 7f366559a640 10 monclient(hunting): _negotiate 2024-03-25T17:59:20.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.860+0000 7f366559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:20.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.860+0000 7f366559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:20.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.860+0000 7f366559a640 10 monclient(hunting): my global_id is 4364 2024-03-25T17:59:20.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.860+0000 7f366559a640 1 -- 172.21.15.18:0/3329395044 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3668142520 con 0x7f36680a98f0 2024-03-25T17:59:20.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.862+0000 7f366559a640 1 -- 172.21.15.18:0/3329395044 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3689987833 0 0) 0x7f3650003520 con 0x7f36681430c0 2024-03-25T17:59:20.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.862+0000 7f366559a640 1 -- 172.21.15.18:0/3329395044 >> v1:172.21.15.18:6790/0 conn(0x7f36680a98f0 legacy=0x7f36680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:20.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.862+0000 7f366559a640 1 -- 172.21.15.18:0/3329395044 >> v1:172.21.15.18:6789/0 conn(0x7f366813f710 legacy=0x7f36681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:20.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.862+0000 7f366559a640 10 monclient: _finish_hunting 0 2024-03-25T17:59:20.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.862+0000 7f366559a640 1 monclient: found mon.noname-c 2024-03-25T17:59:20.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.862+0000 7f366559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:59:20.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.862+0000 7f366559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T17:59:20.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.862+0000 7f366559a640 1 -- 172.21.15.18:0/3329395044 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36680b44c0 con 0x7f36681430c0 2024-03-25T17:59:20.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.862+0000 7f366559a640 10 monclient: _finish_auth 0 2024-03-25T17:59:20.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.862+0000 7f366559a640 10 monclient: _check_auth_tickets 2024-03-25T17:59:20.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.862+0000 7f366559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:59:20.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.862+0000 7f366559a640 1 -- 172.21.15.18:0/3329395044 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3660001a10 con 0x7f36681430c0 2024-03-25T17:59:20.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.862+0000 7f366559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:20.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.862+0000 7f366559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:59:20.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.862+0000 7f366559a640 10 monclient: dump: 2024-03-25T17:59:20.864 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:20.864 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:20.864 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:20.864 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:20.864 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:20.864 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:20.864 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:20.865 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:20.865 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:20.865 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:20.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.862+0000 7f366cbef640 5 monclient: authenticate success, global_id 4345 2024-03-25T17:59:20.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.862+0000 7f366cbef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:59:20.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.872+0000 7f366559a640 1 -- 172.21.15.18:0/3329395044 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3660001b70 con 0x7f36681430c0 2024-03-25T17:59:20.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.872+0000 7f366559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:59:20.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.872+0000 7f366559a640 1 -- 172.21.15.18:0/3329395044 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3660001f80 con 0x7f36681430c0 2024-03-25T17:59:20.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.872+0000 7f366559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:20.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.872+0000 7f366559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:59:20.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.872+0000 7f366559a640 10 monclient: dump: 2024-03-25T17:59:20.874 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:20.874 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:20.874 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:20.874 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:20.874 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:20.874 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:20.874 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:20.874 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:20.874 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:20.874 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:20.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.873+0000 7f366cbef640 10 monclient: get_monmap_and_config success 2024-03-25T17:59:20.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.873+0000 7f366cbef640 4 set_mon_vals no callback set 2024-03-25T17:59:20.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.873+0000 7f366cbef640 10 monclient: shutdown 2024-03-25T17:59:20.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.873+0000 7f366cbef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:59:20.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.873+0000 7f366cbef640 1 -- 172.21.15.18:0/3329395044 >> v1:172.21.15.169:6789/0 conn(0x7f36681430c0 legacy=0x7f366814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:20.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.874+0000 7f366cbef640 1 -- 172.21.15.18:0/3329395044 shutdown_connections 2024-03-25T17:59:20.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.874+0000 7f366cbef640 1 -- 172.21.15.18:0/3329395044 >> 172.21.15.18:0/3329395044 conn(0x7f36680b7ea0 msgr2=0x7f36680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:59:20.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.874+0000 7f366cbef640 1 -- 172.21.15.18:0/3329395044 shutdown_connections 2024-03-25T17:59:20.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.874+0000 7f366cbef640 1 -- 172.21.15.18:0/3329395044 wait complete. 2024-03-25T17:59:20.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.875+0000 7f366cbef640 10 monclient: build_initial_monmap 2024-03-25T17:59:20.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.875+0000 7f366cbef640 10 monclient: monmap: 2024-03-25T17:59:20.876 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:59:20.877 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:59:20.877 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:59:20.877 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:59:20.877 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:59:20.877 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:59:20.877 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:59:20.877 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:59:20.877 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:59:20.877 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:20.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.875+0000 7f366cbef640 1 librados: starting msgr at 2024-03-25T17:59:20.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.875+0000 7f366cbef640 1 librados: starting objecter 2024-03-25T17:59:20.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.875+0000 7f366cbef640 1 Processor -- start 2024-03-25T17:59:20.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.875+0000 7f366cbef640 1 -- start start 2024-03-25T17:59:20.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.875+0000 7f366cbef640 1 librados: setting wanted keys 2024-03-25T17:59:20.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.875+0000 7f366cbef640 1 librados: calling monclient init 2024-03-25T17:59:20.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.875+0000 7f366cbef640 10 monclient: init 2024-03-25T17:59:20.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.876+0000 7f366cbef640 10 monclient: _reopen_session rank -1 2024-03-25T17:59:20.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.876+0000 7f366cbef640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:59:20.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.876+0000 7f366cbef640 10 monclient(hunting): picked mon.noname-a con 0x7f36680a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T17:59:20.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.876+0000 7f366cbef640 10 monclient(hunting): picked mon.noname-b con 0x7f366813f710 addr v1:172.21.15.169:6789/0 2024-03-25T17:59:20.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.876+0000 7f366cbef640 10 monclient(hunting): picked mon.noname-c con 0x7f36681430c0 addr v1:172.21.15.18:6790/0 2024-03-25T17:59:20.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.876+0000 7f366cbef640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3668136ec0 con 0x7f36680a98f0 2024-03-25T17:59:20.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.876+0000 7f366cbef640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3668136c20 con 0x7f366813f710 2024-03-25T17:59:20.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.876+0000 7f366cbef640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3668137db0 con 0x7f36681430c0 2024-03-25T17:59:20.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.876+0000 7f366cbef640 10 monclient(hunting): _renew_subs 2024-03-25T17:59:20.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.876+0000 7f366cbef640 10 monclient(hunting): authenticate will time out at 7089.455539s 2024-03-25T17:59:20.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.882+0000 7f366659c640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f36680a98f0 0x7f36681376a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:50830/0 (socket says 172.21.15.18:50830) 2024-03-25T17:59:20.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.882+0000 7f366659c640 1 -- 172.21.15.18:0/4013438556 learned_addr learned my addr 172.21.15.18:0/4013438556 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:59:20.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.888+0000 7f3656ffd640 10 client.?.objecter ms_handle_connect 0x7f36680a98f0 2024-03-25T17:59:20.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.888+0000 7f3656ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:20.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.898+0000 7f3656ffd640 10 client.?.objecter ms_handle_connect 0x7f366813f710 2024-03-25T17:59:20.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.898+0000 7f3656ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:20.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.900+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899091081 0 0) 0x7f3668136ec0 con 0x7f36680a98f0 2024-03-25T17:59:20.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.900+0000 7f3656ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:59:20.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.900+0000 7f3656ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:20.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.900+0000 7f3656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:20.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.900+0000 7f3656ffd640 10 monclient(hunting): my global_id is 4359 2024-03-25T17:59:20.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.900+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3648004eb0 con 0x7f36680a98f0 2024-03-25T17:59:20.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.901+0000 7f3656ffd640 10 client.?.objecter ms_handle_connect 0x7f36681430c0 2024-03-25T17:59:20.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.901+0000 7f3656ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:20.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.912+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447648017 0 0) 0x7f3668136c20 con 0x7f366813f710 2024-03-25T17:59:20.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.912+0000 7f3656ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:59:20.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.912+0000 7f3656ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:20.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.912+0000 7f3656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:20.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.912+0000 7f3656ffd640 10 monclient(hunting): my global_id is 4348 2024-03-25T17:59:20.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.912+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3668136ec0 con 0x7f366813f710 2024-03-25T17:59:20.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.914+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2439126964 0 0) 0x7f3648004eb0 con 0x7f36680a98f0 2024-03-25T17:59:20.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.914+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3668136c20 con 0x7f36680a98f0 2024-03-25T17:59:20.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.914+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3658001a10 con 0x7f36680a98f0 2024-03-25T17:59:20.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.914+0000 7f3656ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:20.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.914+0000 7f3656ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:59:20.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.914+0000 7f3656ffd640 10 monclient(hunting): dump: 2024-03-25T17:59:20.915 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:20.915 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:20.915 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:20.916 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:20.916 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:20.916 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:20.916 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:20.916 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:20.916 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:20.916 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:20.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.914+0000 7f3656ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:59:20.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.916+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162640337 0 0) 0x7f3668137db0 con 0x7f36681430c0 2024-03-25T17:59:20.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.916+0000 7f3656ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:59:20.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.916+0000 7f3656ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:20.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.916+0000 7f3656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:20.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.916+0000 7f3656ffd640 10 monclient(hunting): my global_id is 4367 2024-03-25T17:59:20.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.916+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3648004eb0 con 0x7f36681430c0 2024-03-25T17:59:20.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1323313555 0 0) 0x7f3668136c20 con 0x7f36680a98f0 2024-03-25T17:59:20.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 >> v1:172.21.15.18:6790/0 conn(0x7f36681430c0 legacy=0x7f36681e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:20.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 >> v1:172.21.15.169:6789/0 conn(0x7f366813f710 legacy=0x7f36680b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f3656ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:59:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f3656ffd640 1 monclient: found mon.a 2024-03-25T17:59:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f3656ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:59:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f3656ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3668138b50 con 0x7f36680a98f0 2024-03-25T17:59:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f3656ffd640 10 monclient: _finish_auth 0 2024-03-25T17:59:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f3656ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:59:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f3656ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:59:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f366cbef640 5 monclient: authenticate success, global_id 4359 2024-03-25T17:59:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f366cbef640 10 monclient: _renew_subs 2024-03-25T17:59:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f366cbef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f366cbef640 1 -- 172.21.15.18:0/4013438556 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3668139830 con 0x7f36680a98f0 2024-03-25T17:59:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f366cbef640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:59:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f366cbef640 10 monclient: _renew_subs 2024-03-25T17:59:20.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f366cbef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:20.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f366cbef640 1 -- 172.21.15.18:0/4013438556 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36681369b0 con 0x7f36680a98f0 2024-03-25T17:59:20.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.927+0000 7f366cbef640 1 librados: init done 2024-03-25T17:59:20.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.928+0000 7f366cbef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:20.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.928+0000 7f366cbef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:20.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.928+0000 7f366cbef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:20.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:20.928+0000 7f366cbef640 1 -- 172.21.15.18:0/4013438556 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36680627b0 con 0x7f36680a98f0 2024-03-25T17:59:21.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.255+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3658001c70 con 0x7f36680a98f0 2024-03-25T17:59:21.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.255+0000 7f3656ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:59:21.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.255+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3658003a60 con 0x7f36680a98f0 2024-03-25T17:59:21.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.255+0000 7f3656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:21.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.255+0000 7f3656ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:59:21.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.255+0000 7f3656ffd640 10 monclient: dump: 2024-03-25T17:59:21.257 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:21.257 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:21.257 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:21.257 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:21.257 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:21.257 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:21.257 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:21.257 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:21.257 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:21.257 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:21.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.255+0000 7f3664d99640 4 set_mon_vals no callback set 2024-03-25T17:59:21.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.256+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f3658003c80 con 0x7f36680a98f0 2024-03-25T17:59:21.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.256+0000 7f3665d9b640 10 monclient: get_auth_request con 0x7f364806de00 auth_method 0 2024-03-25T17:59:21.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.257+0000 7f3656ffd640 10 client.4359.objecter ms_handle_connect 0x7f364806de00 2024-03-25T17:59:21.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.262+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f3658054880 con 0x7f36680a98f0 2024-03-25T17:59:21.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.262+0000 7f3656ffd640 10 client.4359.objecter ms_dispatch 0x7f3668000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:59:21.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.262+0000 7f3656ffd640 3 client.4359.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:59:21.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.262+0000 7f3656ffd640 3 client.4359.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:59:21.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.262+0000 7f3656ffd640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-25T17:59:21.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.262+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3658050390 con 0x7f36680a98f0 2024-03-25T17:59:21.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.263+0000 7f3656ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:21.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.263+0000 7f3656ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:59:21.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.483+0000 7f366cbef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:21.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.483+0000 7f366cbef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:21.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.483+0000 7f366cbef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:21.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.483+0000 7f366cbef640 1 -- 172.21.15.18:0/4013438556 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3668069f40 con 0x7f36680a98f0 2024-03-25T17:59:21.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.505+0000 7f3656ffd640 1 -- 172.21.15.18:0/4013438556 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f36580522c0 con 0x7f36680a98f0 2024-03-25T17:59:21.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.505+0000 7f3656ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:21.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.505+0000 7f3656ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:59:21.506 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:59:21.506 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:59:21.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.507+0000 7f366cbef640 10 librados: watch_flush enter 2024-03-25T17:59:21.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.507+0000 7f366cbef640 10 librados: watch_flush exit 2024-03-25T17:59:21.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.507+0000 7f366cbef640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-25T17:59:21.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.507+0000 7f366cbef640 10 client.4359.objecter successfully canceled tick 2024-03-25T17:59:21.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.507+0000 7f366cbef640 1 -- 172.21.15.18:0/4013438556 >> v1:172.21.15.18:6800/3721244451 conn(0x7f364806de00 legacy=0x7f36480702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:21.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.507+0000 7f366cbef640 10 monclient: shutdown 2024-03-25T17:59:21.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.507+0000 7f366cbef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:59:21.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.507+0000 7f366cbef640 1 -- 172.21.15.18:0/4013438556 >> v1:172.21.15.18:6789/0 conn(0x7f36680a98f0 legacy=0x7f36681376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:21.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.507+0000 7f366cbef640 1 -- 172.21.15.18:0/4013438556 shutdown_connections 2024-03-25T17:59:21.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.507+0000 7f366cbef640 1 -- 172.21.15.18:0/4013438556 >> 172.21.15.18:0/4013438556 conn(0x7f36680b7ea0 msgr2=0x7f366813ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:59:21.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.507+0000 7f366cbef640 1 -- 172.21.15.18:0/4013438556 shutdown_connections 2024-03-25T17:59:21.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.507+0000 7f366cbef640 1 -- 172.21.15.18:0/4013438556 wait complete. 2024-03-25T17:59:21.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:21.507+0000 7f366cbef640 1 librados: shutdown 2024-03-25T17:59:21.519 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:59:22.565 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:20.358 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:22.575 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:20.368 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:22.575 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:20.369 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:22.619 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:20.412 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:22.639 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:20.433 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:22.661 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:03:20.411 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:22.665 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:20.458 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:22.669 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:03:20.418 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:22.687 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:03:20.436 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:28.326 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:59:28.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.439+0000 7f99db7a8640 10 monclient: get_monmap_and_config 2024-03-25T17:59:28.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.439+0000 7f99db7a8640 10 monclient: build_initial_monmap 2024-03-25T17:59:28.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.439+0000 7f99db7a8640 10 monclient: monmap: 2024-03-25T17:59:28.440 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:59:28.440 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:28.440 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:59:28.440626+0000 2024-03-25T17:59:28.440 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:59:28.440626+0000 2024-03-25T17:59:28.440 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:59:28.440 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:28.440 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:59:28.441 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:59:28.441 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:59:28.441 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:28.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.439+0000 7f99db7a8640 1 Processor -- start 2024-03-25T17:59:28.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.440+0000 7f99db7a8640 1 -- start start 2024-03-25T17:59:28.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.440+0000 7f99db7a8640 10 monclient: init 2024-03-25T17:59:28.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.440+0000 7f99db7a8640 10 monclient: _reopen_session rank -1 2024-03-25T17:59:28.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.440+0000 7f99db7a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:59:28.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.440+0000 7f99db7a8640 10 monclient(hunting): picked mon.noname-b con 0x7f99d40a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T17:59:28.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.440+0000 7f99db7a8640 10 monclient(hunting): picked mon.noname-c con 0x7f99d413f710 addr v1:172.21.15.169:6789/0 2024-03-25T17:59:28.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.440+0000 7f99db7a8640 10 monclient(hunting): picked mon.noname-a con 0x7f99d41430c0 addr v1:172.21.15.18:6789/0 2024-03-25T17:59:28.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.440+0000 7f99db7a8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d4142520 con 0x7f99d41430c0 2024-03-25T17:59:28.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.440+0000 7f99db7a8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d40b28a0 con 0x7f99d413f710 2024-03-25T17:59:28.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.440+0000 7f99db7a8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d40b2a30 con 0x7f99d40a98f0 2024-03-25T17:59:28.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.440+0000 7f99db7a8640 10 monclient(hunting): _renew_subs 2024-03-25T17:59:28.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.440+0000 7f99db7a8640 10 monclient(hunting): authenticate will time out at 7097.019606s 2024-03-25T17:59:28.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.444+0000 7f99d8d43640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f99d413f710 0x7f99d41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:60526/0 (socket says 172.21.15.18:60526) 2024-03-25T17:59:28.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.444+0000 7f99d8d43640 1 -- 172.21.15.18:0/30819992 learned_addr learned my addr 172.21.15.18:0/30819992 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:59:28.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.465+0000 7f99cbfff640 1 -- 172.21.15.18:0/30819992 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729530647 0 0) 0x7f99d40b28a0 con 0x7f99d413f710 2024-03-25T17:59:28.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.465+0000 7f99cbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:59:28.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.465+0000 7f99cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:28.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.465+0000 7f99cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:28.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.465+0000 7f99cbfff640 10 monclient(hunting): my global_id is 4351 2024-03-25T17:59:28.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.465+0000 7f99cbfff640 1 -- 172.21.15.18:0/30819992 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99c0003520 con 0x7f99d413f710 2024-03-25T17:59:28.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.469+0000 7f99cbfff640 1 -- 172.21.15.18:0/30819992 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740497497 0 0) 0x7f99d40b2a30 con 0x7f99d40a98f0 2024-03-25T17:59:28.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.469+0000 7f99cbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:59:28.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.469+0000 7f99cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:28.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.469+0000 7f99cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:28.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.469+0000 7f99cbfff640 10 monclient(hunting): my global_id is 4370 2024-03-25T17:59:28.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.469+0000 7f99cbfff640 1 -- 172.21.15.18:0/30819992 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d40b28a0 con 0x7f99d40a98f0 2024-03-25T17:59:28.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.470+0000 7f99cbfff640 1 -- 172.21.15.18:0/30819992 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016968307 0 0) 0x7f99d4142520 con 0x7f99d41430c0 2024-03-25T17:59:28.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.470+0000 7f99cbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:59:28.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.470+0000 7f99cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:28.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.470+0000 7f99cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:28.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.470+0000 7f99cbfff640 10 monclient(hunting): my global_id is 4362 2024-03-25T17:59:28.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.470+0000 7f99cbfff640 1 -- 172.21.15.18:0/30819992 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d40b2a30 con 0x7f99d41430c0 2024-03-25T17:59:28.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.471+0000 7f99cbfff640 1 -- 172.21.15.18:0/30819992 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4209009509 0 0) 0x7f99c0003520 con 0x7f99d413f710 2024-03-25T17:59:28.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.471+0000 7f99cbfff640 1 -- 172.21.15.18:0/30819992 >> v1:172.21.15.18:6790/0 conn(0x7f99d40a98f0 legacy=0x7f99d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:28.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.471+0000 7f99cbfff640 1 -- 172.21.15.18:0/30819992 >> v1:172.21.15.18:6789/0 conn(0x7f99d41430c0 legacy=0x7f99d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:28.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.471+0000 7f99cbfff640 10 monclient: _finish_hunting 0 2024-03-25T17:59:28.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.471+0000 7f99cbfff640 1 monclient: found mon.noname-c 2024-03-25T17:59:28.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.471+0000 7f99cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:59:28.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.471+0000 7f99cbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T17:59:28.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.471+0000 7f99cbfff640 1 -- 172.21.15.18:0/30819992 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d40b44c0 con 0x7f99d413f710 2024-03-25T17:59:28.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.471+0000 7f99cbfff640 10 monclient: _finish_auth 0 2024-03-25T17:59:28.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.471+0000 7f99cbfff640 10 monclient: _check_auth_tickets 2024-03-25T17:59:28.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.471+0000 7f99cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:59:28.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.471+0000 7f99db7a8640 5 monclient: authenticate success, global_id 4351 2024-03-25T17:59:28.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.471+0000 7f99db7a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:59:28.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.473+0000 7f99cbfff640 1 -- 172.21.15.18:0/30819992 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f99c4001650 con 0x7f99d413f710 2024-03-25T17:59:28.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.473+0000 7f99cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:28.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.473+0000 7f99cbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:59:28.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.473+0000 7f99cbfff640 10 monclient: dump: 2024-03-25T17:59:28.474 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:28.474 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:28.474 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:28.474 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:28.474 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:28.474 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:28.475 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:28.475 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:28.475 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:28.475 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:28.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.473+0000 7f99db7a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:59:28.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.480+0000 7f99cbfff640 1 -- 172.21.15.18:0/30819992 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99c40018a0 con 0x7f99d413f710 2024-03-25T17:59:28.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.480+0000 7f99cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:59:28.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.480+0000 7f99cbfff640 1 -- 172.21.15.18:0/30819992 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f99c4001c30 con 0x7f99d413f710 2024-03-25T17:59:28.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.480+0000 7f99db7a8640 10 monclient: get_monmap_and_config success 2024-03-25T17:59:28.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.480+0000 7f99db7a8640 4 set_mon_vals no callback set 2024-03-25T17:59:28.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.480+0000 7f99cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:28.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.480+0000 7f99cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:59:28.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.480+0000 7f99cbfff640 10 monclient: dump: 2024-03-25T17:59:28.481 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:28.481 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:28.481 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:28.481 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:28.481 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:28.481 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:28.481 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:28.481 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:28.481 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:28.481 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:28.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.480+0000 7f99db7a8640 10 monclient: shutdown 2024-03-25T17:59:28.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.480+0000 7f99db7a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:59:28.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.480+0000 7f99db7a8640 1 -- 172.21.15.18:0/30819992 >> v1:172.21.15.169:6789/0 conn(0x7f99d413f710 legacy=0x7f99d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:28.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.481+0000 7f99db7a8640 1 -- 172.21.15.18:0/30819992 shutdown_connections 2024-03-25T17:59:28.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.481+0000 7f99db7a8640 1 -- 172.21.15.18:0/30819992 >> 172.21.15.18:0/30819992 conn(0x7f99d40b7ea0 msgr2=0x7f99d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:59:28.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.481+0000 7f99db7a8640 1 -- 172.21.15.18:0/30819992 shutdown_connections 2024-03-25T17:59:28.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.481+0000 7f99db7a8640 1 -- 172.21.15.18:0/30819992 wait complete. 2024-03-25T17:59:28.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.482+0000 7f99db7a8640 10 monclient: build_initial_monmap 2024-03-25T17:59:28.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.482+0000 7f99db7a8640 10 monclient: monmap: 2024-03-25T17:59:28.484 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:59:28.484 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:59:28.484 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:59:28.484 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:59:28.485 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:59:28.485 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:59:28.485 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:59:28.485 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:59:28.485 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:59:28.485 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:28.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.482+0000 7f99db7a8640 1 librados: starting msgr at 2024-03-25T17:59:28.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.482+0000 7f99db7a8640 1 librados: starting objecter 2024-03-25T17:59:28.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.482+0000 7f99db7a8640 1 Processor -- start 2024-03-25T17:59:28.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.482+0000 7f99db7a8640 1 -- start start 2024-03-25T17:59:28.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.482+0000 7f99db7a8640 1 librados: setting wanted keys 2024-03-25T17:59:28.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.482+0000 7f99db7a8640 1 librados: calling monclient init 2024-03-25T17:59:28.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.482+0000 7f99db7a8640 10 monclient: init 2024-03-25T17:59:28.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.482+0000 7f99db7a8640 10 monclient: _reopen_session rank -1 2024-03-25T17:59:28.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.482+0000 7f99db7a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:59:28.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.482+0000 7f99db7a8640 10 monclient(hunting): picked mon.noname-b con 0x7f99d40a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T17:59:28.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.483+0000 7f99db7a8640 10 monclient(hunting): picked mon.noname-c con 0x7f99d413f710 addr v1:172.21.15.18:6790/0 2024-03-25T17:59:28.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.483+0000 7f99db7a8640 10 monclient(hunting): picked mon.noname-a con 0x7f99d41430c0 addr v1:172.21.15.18:6789/0 2024-03-25T17:59:28.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.483+0000 7f99db7a8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d41378f0 con 0x7f99d41430c0 2024-03-25T17:59:28.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.483+0000 7f99db7a8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d4137e10 con 0x7f99d40a98f0 2024-03-25T17:59:28.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.483+0000 7f99db7a8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d4137460 con 0x7f99d413f710 2024-03-25T17:59:28.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.483+0000 7f99db7a8640 10 monclient(hunting): _renew_subs 2024-03-25T17:59:28.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.483+0000 7f99db7a8640 10 monclient(hunting): authenticate will time out at 7097.061717s 2024-03-25T17:59:28.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.488+0000 7f99d9544640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f99d40a98f0 0x7f99d4133000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:60566/0 (socket says 172.21.15.18:60566) 2024-03-25T17:59:28.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.488+0000 7f99d9544640 1 -- 172.21.15.18:0/710965208 learned_addr learned my addr 172.21.15.18:0/710965208 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:59:28.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.489+0000 7f99c9ffb640 10 client.?.objecter ms_handle_connect 0x7f99d40a98f0 2024-03-25T17:59:28.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.489+0000 7f99c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:28.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.502+0000 7f99c9ffb640 10 client.?.objecter ms_handle_connect 0x7f99d41430c0 2024-03-25T17:59:28.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.502+0000 7f99c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:28.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.504+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709869998 0 0) 0x7f99d4137e10 con 0x7f99d40a98f0 2024-03-25T17:59:28.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.504+0000 7f99c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:59:28.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.504+0000 7f99c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:28.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.504+0000 7f99c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:28.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.504+0000 7f99c9ffb640 10 monclient(hunting): my global_id is 4354 2024-03-25T17:59:28.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.504+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99b8004eb0 con 0x7f99d40a98f0 2024-03-25T17:59:28.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.510+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945692198 0 0) 0x7f99d41378f0 con 0x7f99d41430c0 2024-03-25T17:59:28.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.510+0000 7f99c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:59:28.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.510+0000 7f99c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:28.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.510+0000 7f99c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:28.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.510+0000 7f99c9ffb640 10 monclient(hunting): my global_id is 4365 2024-03-25T17:59:28.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.510+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d4137e10 con 0x7f99d41430c0 2024-03-25T17:59:28.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.511+0000 7f99c9ffb640 10 client.?.objecter ms_handle_connect 0x7f99d413f710 2024-03-25T17:59:28.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.511+0000 7f99c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:28.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.525+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890991706 0 0) 0x7f99d4137460 con 0x7f99d413f710 2024-03-25T17:59:28.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.525+0000 7f99c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:59:28.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.525+0000 7f99c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:28.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.525+0000 7f99c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:28.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.525+0000 7f99c9ffb640 10 monclient(hunting): my global_id is 4373 2024-03-25T17:59:28.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.525+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d41378f0 con 0x7f99d413f710 2024-03-25T17:59:28.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.538+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2729046448 0 0) 0x7f99d41378f0 con 0x7f99d413f710 2024-03-25T17:59:28.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.538+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99d4137460 con 0x7f99d413f710 2024-03-25T17:59:28.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.538+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f99c4001a10 con 0x7f99d413f710 2024-03-25T17:59:28.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.538+0000 7f99c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:28.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.538+0000 7f99c9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:59:28.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.538+0000 7f99c9ffb640 10 monclient(hunting): dump: 2024-03-25T17:59:28.540 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:28.540 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:28.540 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:28.540 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:28.540 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:28.540 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:28.540 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:28.540 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:28.540 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:28.540 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:28.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.538+0000 7f99c9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:59:28.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.539+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 184086388 0 0) 0x7f99b8004eb0 con 0x7f99d40a98f0 2024-03-25T17:59:28.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.539+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99d41378f0 con 0x7f99d40a98f0 2024-03-25T17:59:28.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.539+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f99d00019b0 con 0x7f99d40a98f0 2024-03-25T17:59:28.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.539+0000 7f99c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:28.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.539+0000 7f99c9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:59:28.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.539+0000 7f99c9ffb640 10 monclient(hunting): dump: 2024-03-25T17:59:28.542 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:28.542 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:28.542 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:28.542 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:28.542 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:28.542 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:28.542 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:28.542 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:28.542 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:28.542 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:28.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.540+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4174737856 0 0) 0x7f99d4137e10 con 0x7f99d41430c0 2024-03-25T17:59:28.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.540+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99b8004eb0 con 0x7f99d41430c0 2024-03-25T17:59:28.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.540+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f99cc001bd0 con 0x7f99d41430c0 2024-03-25T17:59:28.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.540+0000 7f99c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:28.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.540+0000 7f99c9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:59:28.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.540+0000 7f99c9ffb640 10 monclient(hunting): dump: 2024-03-25T17:59:28.543 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:28.543 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:28.543 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:28.543 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:28.543 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:28.543 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:28.544 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:28.544 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:28.544 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:28.544 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:28.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.547+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2243767647 0 0) 0x7f99b8004eb0 con 0x7f99d41430c0 2024-03-25T17:59:28.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.547+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 >> v1:172.21.15.18:6790/0 conn(0x7f99d413f710 legacy=0x7f99d41d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:28.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.547+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 >> v1:172.21.15.169:6789/0 conn(0x7f99d40a98f0 legacy=0x7f99d4133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:28.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.547+0000 7f99c9ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:59:28.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.547+0000 7f99c9ffb640 1 monclient: found mon.a 2024-03-25T17:59:28.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.547+0000 7f99c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:59:28.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.547+0000 7f99c9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:28.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.547+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d41375f0 con 0x7f99d41430c0 2024-03-25T17:59:28.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.547+0000 7f99c9ffb640 10 monclient: _finish_auth 0 2024-03-25T17:59:28.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.547+0000 7f99c9ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:59:28.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.547+0000 7f99c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:59:28.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.547+0000 7f99db7a8640 5 monclient: authenticate success, global_id 4365 2024-03-25T17:59:28.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.547+0000 7f99db7a8640 10 monclient: _renew_subs 2024-03-25T17:59:28.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.547+0000 7f99db7a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:28.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.547+0000 7f99db7a8640 1 -- 172.21.15.18:0/710965208 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d4139770 con 0x7f99d41430c0 2024-03-25T17:59:28.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.548+0000 7f99db7a8640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:59:28.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.548+0000 7f99db7a8640 10 monclient: _renew_subs 2024-03-25T17:59:28.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.548+0000 7f99db7a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:28.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.548+0000 7f99db7a8640 1 -- 172.21.15.18:0/710965208 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d41382a0 con 0x7f99d41430c0 2024-03-25T17:59:28.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.548+0000 7f99db7a8640 1 librados: init done 2024-03-25T17:59:28.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.548+0000 7f99db7a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:28.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.549+0000 7f99db7a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:28.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.549+0000 7f99db7a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:28.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.549+0000 7f99db7a8640 1 -- 172.21.15.18:0/710965208 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d414f2d0 con 0x7f99d41430c0 2024-03-25T17:59:28.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.874+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99cc002060 con 0x7f99d41430c0 2024-03-25T17:59:28.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.874+0000 7f99c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:59:28.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.874+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f99cc003b10 con 0x7f99d41430c0 2024-03-25T17:59:28.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.874+0000 7f99c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:28.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.874+0000 7f99c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:59:28.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.874+0000 7f99c9ffb640 10 monclient: dump: 2024-03-25T17:59:28.876 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:28.876 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:28.876 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:28.877 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:28.877 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:28.877 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:28.877 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:28.877 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:28.877 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:28.877 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:28.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.874+0000 7f99cb7fe640 4 set_mon_vals no callback set 2024-03-25T17:59:28.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.874+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f99cc019b20 con 0x7f99d41430c0 2024-03-25T17:59:28.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.875+0000 7f99d9544640 10 monclient: get_auth_request con 0x7f99b806e080 auth_method 0 2024-03-25T17:59:28.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.875+0000 7f99c9ffb640 10 client.4365.objecter ms_handle_connect 0x7f99b806e080 2024-03-25T17:59:28.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.882+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f99cc053880 con 0x7f99d41430c0 2024-03-25T17:59:28.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.882+0000 7f99c9ffb640 10 client.4365.objecter ms_dispatch 0x7f99d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:59:28.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.882+0000 7f99c9ffb640 3 client.4365.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:59:28.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.882+0000 7f99c9ffb640 3 client.4365.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:59:28.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.882+0000 7f99c9ffb640 20 client.4365.objecter dump_active .. 0 homeless 2024-03-25T17:59:28.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.882+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f99cc04f1c0 con 0x7f99d41430c0 2024-03-25T17:59:28.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.882+0000 7f99c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:28.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:28.882+0000 7f99c9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:59:29.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.120+0000 7f99db7a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:29.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.120+0000 7f99db7a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:29.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.120+0000 7f99db7a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:29.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.120+0000 7f99db7a8640 1 -- 172.21.15.18:0/710965208 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99d40627b0 con 0x7f99d41430c0 2024-03-25T17:59:29.161 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.160+0000 7f99c9ffb640 1 -- 172.21.15.18:0/710965208 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f99cc0512c0 con 0x7f99d41430c0 2024-03-25T17:59:29.161 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.160+0000 7f99c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:29.161 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.160+0000 7f99c9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:59:29.162 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:59:29.162 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:59:29.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.163+0000 7f99db7a8640 10 librados: watch_flush enter 2024-03-25T17:59:29.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.163+0000 7f99db7a8640 10 librados: watch_flush exit 2024-03-25T17:59:29.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.163+0000 7f99db7a8640 20 client.4365.objecter shutdown clearing up homeless session... 2024-03-25T17:59:29.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.163+0000 7f99db7a8640 10 client.4365.objecter successfully canceled tick 2024-03-25T17:59:29.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.163+0000 7f99db7a8640 1 -- 172.21.15.18:0/710965208 >> v1:172.21.15.18:6800/3721244451 conn(0x7f99b806e080 legacy=0x7f99b8070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:29.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.163+0000 7f99db7a8640 10 monclient: shutdown 2024-03-25T17:59:29.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.163+0000 7f99db7a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:59:29.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.163+0000 7f99db7a8640 1 -- 172.21.15.18:0/710965208 >> v1:172.21.15.18:6789/0 conn(0x7f99d41430c0 legacy=0x7f99d41e2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:29.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.164+0000 7f99db7a8640 1 -- 172.21.15.18:0/710965208 shutdown_connections 2024-03-25T17:59:29.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.164+0000 7f99db7a8640 1 -- 172.21.15.18:0/710965208 >> 172.21.15.18:0/710965208 conn(0x7f99d40b7ea0 msgr2=0x7f99d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:59:29.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.164+0000 7f99db7a8640 1 -- 172.21.15.18:0/710965208 shutdown_connections 2024-03-25T17:59:29.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.164+0000 7f99db7a8640 1 -- 172.21.15.18:0/710965208 wait complete. 2024-03-25T17:59:29.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:29.164+0000 7f99db7a8640 1 librados: shutdown 2024-03-25T17:59:29.178 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:59:35.986 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:59:36.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.100+0000 7fc2f86e2640 10 monclient: get_monmap_and_config 2024-03-25T17:59:36.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.100+0000 7fc2f86e2640 10 monclient: build_initial_monmap 2024-03-25T17:59:36.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.100+0000 7fc2f86e2640 10 monclient: monmap: 2024-03-25T17:59:36.101 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:59:36.101 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:36.101 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:59:36.101572+0000 2024-03-25T17:59:36.101 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:59:36.101572+0000 2024-03-25T17:59:36.101 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:59:36.101 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:36.101 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:59:36.101 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:59:36.102 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:59:36.102 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:36.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.100+0000 7fc2f86e2640 1 Processor -- start 2024-03-25T17:59:36.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.101+0000 7fc2f86e2640 1 -- start start 2024-03-25T17:59:36.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.101+0000 7fc2f86e2640 10 monclient: init 2024-03-25T17:59:36.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.102+0000 7fc2f86e2640 10 monclient: _reopen_session rank -1 2024-03-25T17:59:36.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.102+0000 7fc2f86e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:59:36.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.102+0000 7fc2f86e2640 10 monclient(hunting): picked mon.noname-c con 0x7fc2f00a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T17:59:36.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.102+0000 7fc2f86e2640 10 monclient(hunting): picked mon.noname-a con 0x7fc2f013f710 addr v1:172.21.15.18:6789/0 2024-03-25T17:59:36.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.102+0000 7fc2f86e2640 10 monclient(hunting): picked mon.noname-b con 0x7fc2f01430c0 addr v1:172.21.15.18:6790/0 2024-03-25T17:59:36.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.102+0000 7fc2f86e2640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f0142520 con 0x7fc2f013f710 2024-03-25T17:59:36.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.102+0000 7fc2f86e2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f00b28a0 con 0x7fc2f00a98f0 2024-03-25T17:59:36.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.102+0000 7fc2f86e2640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f00b2a30 con 0x7fc2f01430c0 2024-03-25T17:59:36.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.102+0000 7fc2f86e2640 10 monclient(hunting): _renew_subs 2024-03-25T17:59:36.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.102+0000 7fc2f86e2640 10 monclient(hunting): authenticate will time out at 7104.680794s 2024-03-25T17:59:36.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.114+0000 7fc2f5c7d640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fc2f013f710 0x7fc2f01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:55058/0 (socket says 172.21.15.18:55058) 2024-03-25T17:59:36.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.114+0000 7fc2f5c7d640 1 -- 172.21.15.18:0/2713792966 learned_addr learned my addr 172.21.15.18:0/2713792966 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:59:36.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.126+0000 7fc2f547c640 1 -- 172.21.15.18:0/2713792966 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652542074 0 0) 0x7fc2f00b28a0 con 0x7fc2f00a98f0 2024-03-25T17:59:36.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.126+0000 7fc2f547c640 10 monclient(hunting): _negotiate 2024-03-25T17:59:36.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.126+0000 7fc2f547c640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:36.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.126+0000 7fc2f547c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:36.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.126+0000 7fc2f547c640 10 monclient(hunting): my global_id is 4357 2024-03-25T17:59:36.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.126+0000 7fc2f547c640 1 -- 172.21.15.18:0/2713792966 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2dc003520 con 0x7fc2f00a98f0 2024-03-25T17:59:36.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.135+0000 7fc2f547c640 1 -- 172.21.15.18:0/2713792966 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973116711 0 0) 0x7fc2f0142520 con 0x7fc2f013f710 2024-03-25T17:59:36.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.135+0000 7fc2f547c640 10 monclient(hunting): _negotiate 2024-03-25T17:59:36.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.135+0000 7fc2f547c640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:36.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.135+0000 7fc2f547c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:36.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.135+0000 7fc2f547c640 10 monclient(hunting): my global_id is 4368 2024-03-25T17:59:36.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.135+0000 7fc2f547c640 1 -- 172.21.15.18:0/2713792966 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2f00b28a0 con 0x7fc2f013f710 2024-03-25T17:59:36.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.154+0000 7fc2f547c640 1 -- 172.21.15.18:0/2713792966 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924082994 0 0) 0x7fc2f00b2a30 con 0x7fc2f01430c0 2024-03-25T17:59:36.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.154+0000 7fc2f547c640 10 monclient(hunting): _negotiate 2024-03-25T17:59:36.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.154+0000 7fc2f547c640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:36.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.154+0000 7fc2f547c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:36.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.154+0000 7fc2f547c640 10 monclient(hunting): my global_id is 4376 2024-03-25T17:59:36.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.154+0000 7fc2f547c640 1 -- 172.21.15.18:0/2713792966 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2f0142520 con 0x7fc2f01430c0 2024-03-25T17:59:36.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f547c640 1 -- 172.21.15.18:0/2713792966 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 565705152 0 0) 0x7fc2dc003520 con 0x7fc2f00a98f0 2024-03-25T17:59:36.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f547c640 1 -- 172.21.15.18:0/2713792966 >> v1:172.21.15.18:6790/0 conn(0x7fc2f01430c0 legacy=0x7fc2f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:36.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f547c640 1 -- 172.21.15.18:0/2713792966 >> v1:172.21.15.18:6789/0 conn(0x7fc2f013f710 legacy=0x7fc2f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:36.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f547c640 10 monclient: _finish_hunting 0 2024-03-25T17:59:36.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f547c640 1 monclient: found mon.noname-c 2024-03-25T17:59:36.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f547c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:59:36.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f547c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T17:59:36.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f547c640 1 -- 172.21.15.18:0/2713792966 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f00b44c0 con 0x7fc2f00a98f0 2024-03-25T17:59:36.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f547c640 10 monclient: _finish_auth 0 2024-03-25T17:59:36.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f547c640 10 monclient: _check_auth_tickets 2024-03-25T17:59:36.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f547c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:59:36.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f547c640 1 -- 172.21.15.18:0/2713792966 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc2ec001690 con 0x7fc2f00a98f0 2024-03-25T17:59:36.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f86e2640 5 monclient: authenticate success, global_id 4357 2024-03-25T17:59:36.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f86e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:59:36.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f547c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:36.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f547c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:59:36.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f547c640 10 monclient: dump: 2024-03-25T17:59:36.160 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:36.160 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:36.160 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:36.160 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:36.160 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:36.160 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:36.160 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:36.160 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:36.160 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:36.160 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:36.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.157+0000 7fc2f86e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:59:36.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.170+0000 7fc2f547c640 1 -- 172.21.15.18:0/2713792966 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2ec0017f0 con 0x7fc2f00a98f0 2024-03-25T17:59:36.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.170+0000 7fc2f547c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:59:36.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.170+0000 7fc2f547c640 1 -- 172.21.15.18:0/2713792966 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc2ec001c00 con 0x7fc2f00a98f0 2024-03-25T17:59:36.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.170+0000 7fc2f547c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:36.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.170+0000 7fc2f547c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:59:36.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.170+0000 7fc2f547c640 10 monclient: dump: 2024-03-25T17:59:36.171 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:36.171 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:36.171 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:36.171 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:36.171 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:36.171 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:36.172 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:36.172 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:36.172 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:36.172 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:36.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.170+0000 7fc2f86e2640 10 monclient: get_monmap_and_config success 2024-03-25T17:59:36.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.170+0000 7fc2f86e2640 4 set_mon_vals no callback set 2024-03-25T17:59:36.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.171+0000 7fc2f86e2640 10 monclient: shutdown 2024-03-25T17:59:36.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.171+0000 7fc2f86e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:59:36.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.171+0000 7fc2f86e2640 1 -- 172.21.15.18:0/2713792966 >> v1:172.21.15.169:6789/0 conn(0x7fc2f00a98f0 legacy=0x7fc2f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:36.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.171+0000 7fc2f86e2640 1 -- 172.21.15.18:0/2713792966 shutdown_connections 2024-03-25T17:59:36.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.171+0000 7fc2f86e2640 1 -- 172.21.15.18:0/2713792966 >> 172.21.15.18:0/2713792966 conn(0x7fc2f00b7ea0 msgr2=0x7fc2f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:59:36.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.171+0000 7fc2f86e2640 1 -- 172.21.15.18:0/2713792966 shutdown_connections 2024-03-25T17:59:36.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.171+0000 7fc2f86e2640 1 -- 172.21.15.18:0/2713792966 wait complete. 2024-03-25T17:59:36.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.172+0000 7fc2f86e2640 10 monclient: build_initial_monmap 2024-03-25T17:59:36.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.172+0000 7fc2f86e2640 10 monclient: monmap: 2024-03-25T17:59:36.175 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:59:36.175 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:59:36.175 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:59:36.175 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:59:36.175 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:59:36.175 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:59:36.175 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:59:36.175 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:59:36.175 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:59:36.175 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:36.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.172+0000 7fc2f86e2640 1 librados: starting msgr at 2024-03-25T17:59:36.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.172+0000 7fc2f86e2640 1 librados: starting objecter 2024-03-25T17:59:36.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.172+0000 7fc2f86e2640 1 Processor -- start 2024-03-25T17:59:36.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.173+0000 7fc2f86e2640 1 -- start start 2024-03-25T17:59:36.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.173+0000 7fc2f86e2640 1 librados: setting wanted keys 2024-03-25T17:59:36.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.173+0000 7fc2f86e2640 1 librados: calling monclient init 2024-03-25T17:59:36.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.173+0000 7fc2f86e2640 10 monclient: init 2024-03-25T17:59:36.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.173+0000 7fc2f86e2640 10 monclient: _reopen_session rank -1 2024-03-25T17:59:36.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.173+0000 7fc2f86e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:59:36.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.173+0000 7fc2f86e2640 10 monclient(hunting): picked mon.noname-b con 0x7fc2f00a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T17:59:36.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.173+0000 7fc2f86e2640 10 monclient(hunting): picked mon.noname-c con 0x7fc2f013f710 addr v1:172.21.15.18:6790/0 2024-03-25T17:59:36.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.173+0000 7fc2f86e2640 10 monclient(hunting): picked mon.noname-a con 0x7fc2f01430c0 addr v1:172.21.15.18:6789/0 2024-03-25T17:59:36.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.173+0000 7fc2f86e2640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f0135590 con 0x7fc2f01430c0 2024-03-25T17:59:36.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.173+0000 7fc2f86e2640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f0136e50 con 0x7fc2f00a98f0 2024-03-25T17:59:36.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.173+0000 7fc2f86e2640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f0135070 con 0x7fc2f013f710 2024-03-25T17:59:36.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.173+0000 7fc2f86e2640 10 monclient(hunting): _renew_subs 2024-03-25T17:59:36.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.173+0000 7fc2f86e2640 10 monclient(hunting): authenticate will time out at 7104.752470s 2024-03-25T17:59:36.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.180+0000 7fc2f6c7f640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fc2f01430c0 0x7fc2f01e3010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:55124/0 (socket says 172.21.15.18:55124) 2024-03-25T17:59:36.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.180+0000 7fc2f6c7f640 1 -- 172.21.15.18:0/238295578 learned_addr learned my addr 172.21.15.18:0/238295578 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:59:36.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.180+0000 7fc2e6ffd640 10 client.?.objecter ms_handle_connect 0x7fc2f01430c0 2024-03-25T17:59:36.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.180+0000 7fc2e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:36.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.194+0000 7fc2e6ffd640 10 client.?.objecter ms_handle_connect 0x7fc2f00a98f0 2024-03-25T17:59:36.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.194+0000 7fc2e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:36.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.198+0000 7fc2e6ffd640 10 client.?.objecter ms_handle_connect 0x7fc2f013f710 2024-03-25T17:59:36.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.198+0000 7fc2e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:36.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.205+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751203017 0 0) 0x7fc2f0136e50 con 0x7fc2f00a98f0 2024-03-25T17:59:36.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.205+0000 7fc2e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:59:36.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.205+0000 7fc2e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:36.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.205+0000 7fc2e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:36.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.205+0000 7fc2e6ffd640 10 monclient(hunting): my global_id is 4360 2024-03-25T17:59:36.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.205+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2d4004eb0 con 0x7fc2f00a98f0 2024-03-25T17:59:36.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.207+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883696045 0 0) 0x7fc2f0135590 con 0x7fc2f01430c0 2024-03-25T17:59:36.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.207+0000 7fc2e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:59:36.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.207+0000 7fc2e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:36.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.207+0000 7fc2e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:36.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.207+0000 7fc2e6ffd640 10 monclient(hunting): my global_id is 4371 2024-03-25T17:59:36.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.207+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2f0136e50 con 0x7fc2f01430c0 2024-03-25T17:59:36.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.212+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567243675 0 0) 0x7fc2f0135070 con 0x7fc2f013f710 2024-03-25T17:59:36.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.212+0000 7fc2e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:59:36.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.212+0000 7fc2e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:36.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.212+0000 7fc2e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:36.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.212+0000 7fc2e6ffd640 10 monclient(hunting): my global_id is 4379 2024-03-25T17:59:36.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.212+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2f0135590 con 0x7fc2f013f710 2024-03-25T17:59:36.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.233+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2607482739 0 0) 0x7fc2f0136e50 con 0x7fc2f01430c0 2024-03-25T17:59:36.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.234+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2f0135070 con 0x7fc2f01430c0 2024-03-25T17:59:36.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.234+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc2e8001c50 con 0x7fc2f01430c0 2024-03-25T17:59:36.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.234+0000 7fc2e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:36.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.234+0000 7fc2e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:59:36.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.234+0000 7fc2e6ffd640 10 monclient(hunting): dump: 2024-03-25T17:59:36.235 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:36.235 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:36.235 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:36.235 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:36.235 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:36.235 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:36.235 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:36.235 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:36.236 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:36.236 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:36.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.234+0000 7fc2e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:59:36.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.234+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2504715233 0 0) 0x7fc2d4004eb0 con 0x7fc2f00a98f0 2024-03-25T17:59:36.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.234+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2f0136e50 con 0x7fc2f00a98f0 2024-03-25T17:59:36.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.234+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc2ec001b00 con 0x7fc2f00a98f0 2024-03-25T17:59:36.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.234+0000 7fc2e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:36.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.234+0000 7fc2e6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:59:36.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.234+0000 7fc2e6ffd640 10 monclient(hunting): dump: 2024-03-25T17:59:36.237 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:36.237 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:36.237 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:36.237 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:36.237 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:36.237 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:36.237 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:36.237 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:36.237 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:36.238 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:36.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1726071754 0 0) 0x7fc2f0136e50 con 0x7fc2f00a98f0 2024-03-25T17:59:36.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 >> v1:172.21.15.18:6790/0 conn(0x7fc2f013f710 legacy=0x7fc2f01d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:36.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 >> v1:172.21.15.18:6789/0 conn(0x7fc2f01430c0 legacy=0x7fc2f01e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:36.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:59:36.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2e6ffd640 1 monclient: found mon.b 2024-03-25T17:59:36.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:59:36.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2e6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:36.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f0135d80 con 0x7fc2f00a98f0 2024-03-25T17:59:36.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2e6ffd640 10 monclient: _finish_auth 0 2024-03-25T17:59:36.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2e6ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:59:36.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:59:36.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2f86e2640 5 monclient: authenticate success, global_id 4360 2024-03-25T17:59:36.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2f86e2640 10 monclient: _renew_subs 2024-03-25T17:59:36.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2f86e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:36.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2f86e2640 1 -- 172.21.15.18:0/238295578 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f0131660 con 0x7fc2f00a98f0 2024-03-25T17:59:36.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2f86e2640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:59:36.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2f86e2640 10 monclient: _renew_subs 2024-03-25T17:59:36.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2f86e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:36.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2f86e2640 1 -- 172.21.15.18:0/238295578 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f00b78a0 con 0x7fc2f00a98f0 2024-03-25T17:59:36.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.242+0000 7fc2f86e2640 1 librados: init done 2024-03-25T17:59:36.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.243+0000 7fc2f86e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:36.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.243+0000 7fc2f86e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:36.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.243+0000 7fc2f86e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:36.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.243+0000 7fc2f86e2640 1 -- 172.21.15.18:0/238295578 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f0135f70 con 0x7fc2f00a98f0 2024-03-25T17:59:36.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.566+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2ec0038a0 con 0x7fc2f00a98f0 2024-03-25T17:59:36.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.566+0000 7fc2e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:59:36.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.566+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc2ec003a10 con 0x7fc2f00a98f0 2024-03-25T17:59:36.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.566+0000 7fc2e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:36.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.566+0000 7fc2e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:59:36.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.566+0000 7fc2e6ffd640 10 monclient: dump: 2024-03-25T17:59:36.567 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:36.567 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:36.567 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:36.567 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:36.567 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:36.567 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:36.567 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:36.567 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:36.567 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:36.567 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:36.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.566+0000 7fc2f4c7b640 4 set_mon_vals no callback set 2024-03-25T17:59:36.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.566+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fc2ec0199e0 con 0x7fc2f00a98f0 2024-03-25T17:59:36.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.567+0000 7fc2f5c7d640 10 monclient: get_auth_request con 0x7fc2d406dfb0 auth_method 0 2024-03-25T17:59:36.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.567+0000 7fc2e6ffd640 10 client.4360.objecter ms_handle_connect 0x7fc2d406dfb0 2024-03-25T17:59:36.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.570+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fc2ec053510 con 0x7fc2f00a98f0 2024-03-25T17:59:36.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.570+0000 7fc2e6ffd640 10 client.4360.objecter ms_dispatch 0x7fc2f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:59:36.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.570+0000 7fc2e6ffd640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:59:36.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.570+0000 7fc2e6ffd640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:59:36.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.570+0000 7fc2e6ffd640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-25T17:59:36.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.570+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2ec04f020 con 0x7fc2f00a98f0 2024-03-25T17:59:36.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.570+0000 7fc2e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:36.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.570+0000 7fc2e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:59:36.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.821+0000 7fc2f86e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:36.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.821+0000 7fc2f86e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:36.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.821+0000 7fc2f86e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:36.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.821+0000 7fc2f86e2640 1 -- 172.21.15.18:0/238295578 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2f0135200 con 0x7fc2f00a98f0 2024-03-25T17:59:36.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.849+0000 7fc2e6ffd640 1 -- 172.21.15.18:0/238295578 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fc2ec050f50 con 0x7fc2f00a98f0 2024-03-25T17:59:36.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.849+0000 7fc2e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:36.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.849+0000 7fc2e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:59:36.850 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:59:36.850 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:59:36.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.851+0000 7fc2f86e2640 10 librados: watch_flush enter 2024-03-25T17:59:36.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.851+0000 7fc2f86e2640 10 librados: watch_flush exit 2024-03-25T17:59:36.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.851+0000 7fc2f86e2640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-25T17:59:36.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.851+0000 7fc2f86e2640 10 client.4360.objecter successfully canceled tick 2024-03-25T17:59:36.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.851+0000 7fc2f86e2640 1 -- 172.21.15.18:0/238295578 >> v1:172.21.15.18:6800/3721244451 conn(0x7fc2d406dfb0 legacy=0x7fc2d4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:36.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.851+0000 7fc2f86e2640 10 monclient: shutdown 2024-03-25T17:59:36.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.851+0000 7fc2f86e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:59:36.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.851+0000 7fc2f86e2640 1 -- 172.21.15.18:0/238295578 >> v1:172.21.15.169:6789/0 conn(0x7fc2f00a98f0 legacy=0x7fc2f0130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:36.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.851+0000 7fc2f86e2640 1 -- 172.21.15.18:0/238295578 shutdown_connections 2024-03-25T17:59:36.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.852+0000 7fc2f86e2640 1 -- 172.21.15.18:0/238295578 >> 172.21.15.18:0/238295578 conn(0x7fc2f00b7ea0 msgr2=0x7fc2f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:59:36.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.852+0000 7fc2f86e2640 1 -- 172.21.15.18:0/238295578 shutdown_connections 2024-03-25T17:59:36.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.852+0000 7fc2f86e2640 1 -- 172.21.15.18:0/238295578 wait complete. 2024-03-25T17:59:36.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:36.852+0000 7fc2f86e2640 1 librados: shutdown 2024-03-25T17:59:36.864 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:59:42.622 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:40.415 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:42.643 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:40.436 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:42.667 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:03:40.460 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:42.685 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:03:40.434 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:42.687 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:40.480 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:42.688 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:03:40.437 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:42.699 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:40.492 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:42.721 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:03:40.470 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:42.722 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:03:40.515 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T17:59:43.671 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:59:43.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.771+0000 7f3e01dbc640 10 monclient: get_monmap_and_config 2024-03-25T17:59:43.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.771+0000 7f3e01dbc640 10 monclient: build_initial_monmap 2024-03-25T17:59:43.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.771+0000 7f3e01dbc640 10 monclient: monmap: 2024-03-25T17:59:43.772 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:59:43.772 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:43.772 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:59:43.772278+0000 2024-03-25T17:59:43.772 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:59:43.772278+0000 2024-03-25T17:59:43.772 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:59:43.772 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:43.772 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:59:43.772 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:59:43.772 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:59:43.772 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:43.774 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.772+0000 7f3e01dbc640 1 Processor -- start 2024-03-25T17:59:43.774 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.772+0000 7f3e01dbc640 1 -- start start 2024-03-25T17:59:43.774 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.772+0000 7f3e01dbc640 10 monclient: init 2024-03-25T17:59:43.774 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.773+0000 7f3e01dbc640 10 monclient: _reopen_session rank -1 2024-03-25T17:59:43.774 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.773+0000 7f3e01dbc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:59:43.774 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.773+0000 7f3e01dbc640 10 monclient(hunting): picked mon.noname-c con 0x7f3dfc0a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T17:59:43.774 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.773+0000 7f3e01dbc640 10 monclient(hunting): picked mon.noname-b con 0x7f3dfc13f710 addr v1:172.21.15.18:6790/0 2024-03-25T17:59:43.774 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.773+0000 7f3e01dbc640 10 monclient(hunting): picked mon.noname-a con 0x7f3dfc1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T17:59:43.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.773+0000 7f3e01dbc640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dfc142520 con 0x7f3dfc1430c0 2024-03-25T17:59:43.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.773+0000 7f3e01dbc640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dfc0b28a0 con 0x7f3dfc0a98f0 2024-03-25T17:59:43.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.773+0000 7f3e01dbc640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dfc0b2a30 con 0x7f3dfc13f710 2024-03-25T17:59:43.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.773+0000 7f3e01dbc640 10 monclient(hunting): _renew_subs 2024-03-25T17:59:43.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.773+0000 7f3e01dbc640 10 monclient(hunting): authenticate will time out at 7112.351009s 2024-03-25T17:59:43.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.773+0000 7f3dfaffd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f3dfc13f710 0x7f3dfc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:39736/0 (socket says 172.21.15.18:39736) 2024-03-25T17:59:43.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.773+0000 7f3dfaffd640 1 -- 172.21.15.18:0/1539085225 learned_addr learned my addr 172.21.15.18:0/1539085225 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:59:43.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.800+0000 7f3dfa7fc640 1 -- 172.21.15.18:0/1539085225 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899901774 0 0) 0x7f3dfc0b28a0 con 0x7f3dfc0a98f0 2024-03-25T17:59:43.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.800+0000 7f3dfa7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:59:43.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.800+0000 7f3dfa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:43.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.800+0000 7f3dfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:43.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.800+0000 7f3dfa7fc640 10 monclient(hunting): my global_id is 4363 2024-03-25T17:59:43.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.800+0000 7f3dfa7fc640 1 -- 172.21.15.18:0/1539085225 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de8003520 con 0x7f3dfc0a98f0 2024-03-25T17:59:43.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.819+0000 7f3dfa7fc640 1 -- 172.21.15.18:0/1539085225 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4037181659 0 0) 0x7f3de8003520 con 0x7f3dfc0a98f0 2024-03-25T17:59:43.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.819+0000 7f3dfa7fc640 1 -- 172.21.15.18:0/1539085225 >> v1:172.21.15.18:6790/0 conn(0x7f3dfc13f710 legacy=0x7f3dfc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:43.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.819+0000 7f3dfa7fc640 1 -- 172.21.15.18:0/1539085225 >> v1:172.21.15.18:6789/0 conn(0x7f3dfc1430c0 legacy=0x7f3dfc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:43.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.819+0000 7f3dfa7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:59:43.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.819+0000 7f3dfa7fc640 1 monclient: found mon.noname-c 2024-03-25T17:59:43.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.820+0000 7f3dfa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:59:43.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.820+0000 7f3dfa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T17:59:43.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.820+0000 7f3dfa7fc640 1 -- 172.21.15.18:0/1539085225 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc0b44c0 con 0x7f3dfc0a98f0 2024-03-25T17:59:43.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.820+0000 7f3dfa7fc640 10 monclient: _finish_auth 0 2024-03-25T17:59:43.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.820+0000 7f3dfa7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:59:43.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.820+0000 7f3dfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:59:43.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.820+0000 7f3dfa7fc640 1 -- 172.21.15.18:0/1539085225 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3dec001690 con 0x7f3dfc0a98f0 2024-03-25T17:59:43.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.820+0000 7f3e01dbc640 5 monclient: authenticate success, global_id 4363 2024-03-25T17:59:43.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.820+0000 7f3e01dbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:59:43.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.820+0000 7f3dfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:43.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.820+0000 7f3dfa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:59:43.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.820+0000 7f3dfa7fc640 10 monclient: dump: 2024-03-25T17:59:43.821 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:43.821 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:43.821 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:43.821 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:43.821 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:43.821 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:43.822 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:43.822 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:43.822 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:43.822 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:43.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.820+0000 7f3e01dbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:59:43.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.831+0000 7f3dfa7fc640 1 -- 172.21.15.18:0/1539085225 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3dec0017f0 con 0x7f3dfc0a98f0 2024-03-25T17:59:43.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.831+0000 7f3dfa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:59:43.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.831+0000 7f3dfa7fc640 1 -- 172.21.15.18:0/1539085225 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3dec001c00 con 0x7f3dfc0a98f0 2024-03-25T17:59:43.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.831+0000 7f3e01dbc640 10 monclient: get_monmap_and_config success 2024-03-25T17:59:43.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.831+0000 7f3e01dbc640 4 set_mon_vals no callback set 2024-03-25T17:59:43.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.831+0000 7f3dfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:43.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.831+0000 7f3dfa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:59:43.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.831+0000 7f3dfa7fc640 10 monclient: dump: 2024-03-25T17:59:43.831 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:43.832 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:43.832 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:43.832 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:43.832 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:43.832 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:43.832 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:43.832 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:43.832 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:43.832 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:43.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.832+0000 7f3e01dbc640 10 monclient: shutdown 2024-03-25T17:59:43.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.832+0000 7f3e01dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:59:43.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.832+0000 7f3e01dbc640 1 -- 172.21.15.18:0/1539085225 >> v1:172.21.15.169:6789/0 conn(0x7f3dfc0a98f0 legacy=0x7f3dfc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:43.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.832+0000 7f3e01dbc640 1 -- 172.21.15.18:0/1539085225 shutdown_connections 2024-03-25T17:59:43.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.832+0000 7f3e01dbc640 1 -- 172.21.15.18:0/1539085225 >> 172.21.15.18:0/1539085225 conn(0x7f3dfc0b7ea0 msgr2=0x7f3dfc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:59:43.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.832+0000 7f3e01dbc640 1 -- 172.21.15.18:0/1539085225 shutdown_connections 2024-03-25T17:59:43.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.832+0000 7f3e01dbc640 1 -- 172.21.15.18:0/1539085225 wait complete. 2024-03-25T17:59:43.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.833+0000 7f3e01dbc640 10 monclient: build_initial_monmap 2024-03-25T17:59:43.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.833+0000 7f3e01dbc640 10 monclient: monmap: 2024-03-25T17:59:43.834 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:59:43.834 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:59:43.834 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:59:43.834 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:59:43.834 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:59:43.835 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:59:43.835 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:59:43.835 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:59:43.835 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:59:43.835 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:43.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.833+0000 7f3e01dbc640 1 librados: starting msgr at 2024-03-25T17:59:43.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.833+0000 7f3e01dbc640 1 librados: starting objecter 2024-03-25T17:59:43.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.833+0000 7f3e01dbc640 1 Processor -- start 2024-03-25T17:59:43.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.834+0000 7f3e01dbc640 1 -- start start 2024-03-25T17:59:43.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.834+0000 7f3e01dbc640 1 librados: setting wanted keys 2024-03-25T17:59:43.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.834+0000 7f3e01dbc640 1 librados: calling monclient init 2024-03-25T17:59:43.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.834+0000 7f3e01dbc640 10 monclient: init 2024-03-25T17:59:43.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.834+0000 7f3e01dbc640 10 monclient: _reopen_session rank -1 2024-03-25T17:59:43.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.834+0000 7f3e01dbc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:59:43.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.834+0000 7f3e01dbc640 10 monclient(hunting): picked mon.noname-c con 0x7f3dfc0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T17:59:43.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.834+0000 7f3e01dbc640 10 monclient(hunting): picked mon.noname-b con 0x7f3dfc13f710 addr v1:172.21.15.169:6789/0 2024-03-25T17:59:43.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.834+0000 7f3e01dbc640 10 monclient(hunting): picked mon.noname-a con 0x7f3dfc1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T17:59:43.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.834+0000 7f3e01dbc640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dfc13be70 con 0x7f3dfc1430c0 2024-03-25T17:59:43.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.834+0000 7f3e01dbc640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dfc131660 con 0x7f3dfc13f710 2024-03-25T17:59:43.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.834+0000 7f3e01dbc640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dfc1317f0 con 0x7f3dfc0a98f0 2024-03-25T17:59:43.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.834+0000 7f3e01dbc640 10 monclient(hunting): _renew_subs 2024-03-25T17:59:43.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.834+0000 7f3e01dbc640 10 monclient(hunting): authenticate will time out at 7112.412502s 2024-03-25T17:59:43.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.847+0000 7f3dfbfff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f3dfc1430c0 0x7f3dfc1e2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:59546/0 (socket says 172.21.15.18:59546) 2024-03-25T17:59:43.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.847+0000 7f3dfbfff640 1 -- 172.21.15.18:0/1071317359 learned_addr learned my addr 172.21.15.18:0/1071317359 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:59:43.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.847+0000 7f3df3fff640 10 client.?.objecter ms_handle_connect 0x7f3dfc1430c0 2024-03-25T17:59:43.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.847+0000 7f3df3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:43.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.871+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127802033 0 0) 0x7f3dfc13be70 con 0x7f3dfc1430c0 2024-03-25T17:59:43.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.871+0000 7f3df3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:59:43.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.871+0000 7f3df3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:43.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.871+0000 7f3df3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:43.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.871+0000 7f3df3fff640 10 monclient(hunting): my global_id is 4377 2024-03-25T17:59:43.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.871+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ddc004eb0 con 0x7f3dfc1430c0 2024-03-25T17:59:43.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.873+0000 7f3df3fff640 10 client.?.objecter ms_handle_connect 0x7f3dfc13f710 2024-03-25T17:59:43.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.873+0000 7f3df3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:43.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.879+0000 7f3df3fff640 10 client.?.objecter ms_handle_connect 0x7f3dfc0a98f0 2024-03-25T17:59:43.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.879+0000 7f3df3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:43.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.890+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587325031 0 0) 0x7f3dfc131660 con 0x7f3dfc13f710 2024-03-25T17:59:43.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.890+0000 7f3df3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:59:43.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.890+0000 7f3df3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:43.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.890+0000 7f3df3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:43.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.890+0000 7f3df3fff640 10 monclient(hunting): my global_id is 4366 2024-03-25T17:59:43.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.891+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dfc13be70 con 0x7f3dfc13f710 2024-03-25T17:59:43.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.904+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722895854 0 0) 0x7f3dfc1317f0 con 0x7f3dfc0a98f0 2024-03-25T17:59:43.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.904+0000 7f3df3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:59:43.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.904+0000 7f3df3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:43.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.904+0000 7f3df3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:43.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.904+0000 7f3df3fff640 10 monclient(hunting): my global_id is 4385 2024-03-25T17:59:43.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.904+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dfc131660 con 0x7f3dfc0a98f0 2024-03-25T17:59:43.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.909+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2203048938 0 0) 0x7f3dfc13be70 con 0x7f3dfc13f710 2024-03-25T17:59:43.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.909+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3dfc1317f0 con 0x7f3dfc13f710 2024-03-25T17:59:43.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.909+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3de40018a0 con 0x7f3dfc13f710 2024-03-25T17:59:43.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.909+0000 7f3df3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:43.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.909+0000 7f3df3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:59:43.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.909+0000 7f3df3fff640 10 monclient(hunting): dump: 2024-03-25T17:59:43.910 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:43.910 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:43.910 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:43.910 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:43.910 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:43.910 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:43.910 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:43.910 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:43.910 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:43.910 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:43.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.909+0000 7f3df3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:59:43.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.919+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1995874763 0 0) 0x7f3dfc131660 con 0x7f3dfc0a98f0 2024-03-25T17:59:43.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.919+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3dfc13be70 con 0x7f3dfc0a98f0 2024-03-25T17:59:43.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.919+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3dec001b00 con 0x7f3dfc0a98f0 2024-03-25T17:59:43.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.919+0000 7f3df3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:43.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.919+0000 7f3df3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:59:43.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.919+0000 7f3df3fff640 10 monclient(hunting): dump: 2024-03-25T17:59:43.919 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:43.919 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:43.920 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:43.920 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:43.920 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:43.920 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:43.920 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:43.920 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:43.920 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:43.920 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:43.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.925+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1865771304 0 0) 0x7f3ddc004eb0 con 0x7f3dfc1430c0 2024-03-25T17:59:43.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.925+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3dfc131660 con 0x7f3dfc1430c0 2024-03-25T17:59:43.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.925+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3df4001c90 con 0x7f3dfc1430c0 2024-03-25T17:59:43.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.925+0000 7f3df3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:43.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.925+0000 7f3df3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:59:43.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.925+0000 7f3df3fff640 10 monclient(hunting): dump: 2024-03-25T17:59:43.925 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:43.925 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:43.925 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:43.925 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:43.925 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:43.925 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:43.925 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:43.925 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:43.926 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:43.926 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:43.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.929+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4085401739 0 0) 0x7f3dfc1317f0 con 0x7f3dfc13f710 2024-03-25T17:59:43.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.929+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 >> v1:172.21.15.18:6790/0 conn(0x7f3dfc0a98f0 legacy=0x7f3dfc130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:43.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.929+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 >> v1:172.21.15.18:6789/0 conn(0x7f3dfc1430c0 legacy=0x7f3dfc1e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:43.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.929+0000 7f3df3fff640 10 monclient: _finish_hunting 0 2024-03-25T17:59:43.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.929+0000 7f3df3fff640 1 monclient: found mon.b 2024-03-25T17:59:43.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.929+0000 7f3df3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:59:43.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.929+0000 7f3df3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:43.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.929+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc131980 con 0x7f3dfc13f710 2024-03-25T17:59:43.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.929+0000 7f3df3fff640 10 monclient: _finish_auth 0 2024-03-25T17:59:43.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.929+0000 7f3df3fff640 10 monclient: _check_auth_tickets 2024-03-25T17:59:43.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.929+0000 7f3df3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:59:43.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.929+0000 7f3e01dbc640 5 monclient: authenticate success, global_id 4366 2024-03-25T17:59:43.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.929+0000 7f3e01dbc640 10 monclient: _renew_subs 2024-03-25T17:59:43.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.929+0000 7f3e01dbc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:43.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.929+0000 7f3e01dbc640 1 -- 172.21.15.18:0/1071317359 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dfc135c60 con 0x7f3dfc13f710 2024-03-25T17:59:43.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.930+0000 7f3e01dbc640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:59:43.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.930+0000 7f3e01dbc640 10 monclient: _renew_subs 2024-03-25T17:59:43.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.930+0000 7f3e01dbc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:43.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.930+0000 7f3e01dbc640 1 -- 172.21.15.18:0/1071317359 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dfc0b7770 con 0x7f3dfc13f710 2024-03-25T17:59:43.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.930+0000 7f3e01dbc640 1 librados: init done 2024-03-25T17:59:43.932 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.931+0000 7f3e01dbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:43.932 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.931+0000 7f3e01dbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:43.932 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.931+0000 7f3e01dbc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:43.932 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:43.931+0000 7f3e01dbc640 1 -- 172.21.15.18:0/1071317359 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dfc074cc0 con 0x7f3dfc13f710 2024-03-25T17:59:44.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.209+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3de4001cb0 con 0x7f3dfc13f710 2024-03-25T17:59:44.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.209+0000 7f3df3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:59:44.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.209+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3de4004980 con 0x7f3dfc13f710 2024-03-25T17:59:44.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.209+0000 7f3df3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:44.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.209+0000 7f3df9ffb640 4 set_mon_vals no callback set 2024-03-25T17:59:44.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.209+0000 7f3df3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:59:44.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.209+0000 7f3df3fff640 10 monclient: dump: 2024-03-25T17:59:44.209 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:44.210 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:44.210 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:44.210 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:44.210 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:44.210 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:44.210 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:44.210 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:44.210 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:44.210 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:44.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.209+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f3de4004b60 con 0x7f3dfc13f710 2024-03-25T17:59:44.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.210+0000 7f3dfb7fe640 10 monclient: get_auth_request con 0x7f3ddc06e180 auth_method 0 2024-03-25T17:59:44.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.210+0000 7f3df3fff640 10 client.4366.objecter ms_handle_connect 0x7f3ddc06e180 2024-03-25T17:59:44.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.214+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f3de4054680 con 0x7f3dfc13f710 2024-03-25T17:59:44.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.214+0000 7f3df3fff640 10 client.4366.objecter ms_dispatch 0x7f3dfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:59:44.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.214+0000 7f3df3fff640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:59:44.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.214+0000 7f3df3fff640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:59:44.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.214+0000 7f3df3fff640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T17:59:44.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.215+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3de404ffc0 con 0x7f3dfc13f710 2024-03-25T17:59:44.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.215+0000 7f3df3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:44.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.215+0000 7f3df3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:59:44.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.470+0000 7f3e01dbc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:44.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.471+0000 7f3e01dbc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:44.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.471+0000 7f3e01dbc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T17:59:44.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.471+0000 7f3e01dbc640 1 -- 172.21.15.18:0/1071317359 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3dfc1371d0 con 0x7f3dfc13f710 2024-03-25T17:59:44.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.508+0000 7f3df3fff640 1 -- 172.21.15.18:0/1071317359 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f3de40520c0 con 0x7f3dfc13f710 2024-03-25T17:59:44.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.508+0000 7f3df3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:44.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.508+0000 7f3df3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:59:44.509 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:59:44.509 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:59:44.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.511+0000 7f3e01dbc640 10 librados: watch_flush enter 2024-03-25T17:59:44.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.511+0000 7f3e01dbc640 10 librados: watch_flush exit 2024-03-25T17:59:44.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.511+0000 7f3e01dbc640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T17:59:44.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.511+0000 7f3e01dbc640 10 client.4366.objecter successfully canceled tick 2024-03-25T17:59:44.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.511+0000 7f3e01dbc640 1 -- 172.21.15.18:0/1071317359 >> v1:172.21.15.18:6800/3721244451 conn(0x7f3ddc06e180 legacy=0x7f3ddc070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:44.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.511+0000 7f3e01dbc640 10 monclient: shutdown 2024-03-25T17:59:44.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.511+0000 7f3e01dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:59:44.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.511+0000 7f3e01dbc640 1 -- 172.21.15.18:0/1071317359 >> v1:172.21.15.169:6789/0 conn(0x7f3dfc13f710 legacy=0x7f3dfc1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:44.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.511+0000 7f3e01dbc640 1 -- 172.21.15.18:0/1071317359 shutdown_connections 2024-03-25T17:59:44.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.511+0000 7f3e01dbc640 1 -- 172.21.15.18:0/1071317359 >> 172.21.15.18:0/1071317359 conn(0x7f3dfc0b7ea0 msgr2=0x7f3dfc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:59:44.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.511+0000 7f3e01dbc640 1 -- 172.21.15.18:0/1071317359 shutdown_connections 2024-03-25T17:59:44.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.511+0000 7f3e01dbc640 1 -- 172.21.15.18:0/1071317359 wait complete. 2024-03-25T17:59:44.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:44.511+0000 7f3e01dbc640 1 librados: shutdown 2024-03-25T17:59:44.523 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:59:51.330 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:59:51.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.445+0000 7fa9160d3640 10 monclient: get_monmap_and_config 2024-03-25T17:59:51.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.445+0000 7fa9160d3640 10 monclient: build_initial_monmap 2024-03-25T17:59:51.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.445+0000 7fa9160d3640 10 monclient: monmap: 2024-03-25T17:59:51.445 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:59:51.445 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:51.445 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:59:51.445695+0000 2024-03-25T17:59:51.445 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:59:51.445695+0000 2024-03-25T17:59:51.445 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:59:51.445 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:51.445 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:59:51.446 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:59:51.446 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:59:51.446 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:51.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.446+0000 7fa9160d3640 1 Processor -- start 2024-03-25T17:59:51.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.446+0000 7fa9160d3640 1 -- start start 2024-03-25T17:59:51.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.446+0000 7fa9160d3640 10 monclient: init 2024-03-25T17:59:51.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.446+0000 7fa9160d3640 10 monclient: _reopen_session rank -1 2024-03-25T17:59:51.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.446+0000 7fa9160d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:59:51.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.446+0000 7fa9160d3640 10 monclient(hunting): picked mon.noname-b con 0x7fa9100b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T17:59:51.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.446+0000 7fa9160d3640 10 monclient(hunting): picked mon.noname-a con 0x7fa9100b1730 addr v1:172.21.15.18:6789/0 2024-03-25T17:59:51.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.446+0000 7fa9160d3640 10 monclient(hunting): picked mon.noname-c con 0x7fa91014be90 addr v1:172.21.15.169:6789/0 2024-03-25T17:59:51.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.446+0000 7fa9160d3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9100af940 con 0x7fa9100b1730 2024-03-25T17:59:51.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.446+0000 7fa9160d3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9100b0a70 con 0x7fa91014be90 2024-03-25T17:59:51.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.446+0000 7fa9160d3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9100b0d50 con 0x7fa9100b4d60 2024-03-25T17:59:51.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.446+0000 7fa9160d3640 10 monclient(hunting): _renew_subs 2024-03-25T17:59:51.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.446+0000 7fa9160d3640 10 monclient(hunting): authenticate will time out at 7120.024592s 2024-03-25T17:59:51.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.462+0000 7fa90ffff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fa91014be90 0x7fa91014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:32862/0 (socket says 172.21.15.18:32862) 2024-03-25T17:59:51.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.462+0000 7fa90ffff640 1 -- 172.21.15.18:0/408821947 learned_addr learned my addr 172.21.15.18:0/408821947 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:59:51.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.485+0000 7fa90e7fc640 1 -- 172.21.15.18:0/408821947 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1200611176 0 0) 0x7fa9100b0a70 con 0x7fa91014be90 2024-03-25T17:59:51.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.485+0000 7fa90e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:59:51.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.485+0000 7fa90e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:51.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.485+0000 7fa90e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:51.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.485+0000 7fa90e7fc640 10 monclient(hunting): my global_id is 4369 2024-03-25T17:59:51.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.485+0000 7fa90e7fc640 1 -- 172.21.15.18:0/408821947 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8f8003520 con 0x7fa91014be90 2024-03-25T17:59:51.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.488+0000 7fa90e7fc640 1 -- 172.21.15.18:0/408821947 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814745074 0 0) 0x7fa9100b0d50 con 0x7fa9100b4d60 2024-03-25T17:59:51.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.489+0000 7fa90e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:59:51.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.489+0000 7fa90e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:51.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.489+0000 7fa90e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:51.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.489+0000 7fa90e7fc640 10 monclient(hunting): my global_id is 4388 2024-03-25T17:59:51.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.489+0000 7fa90e7fc640 1 -- 172.21.15.18:0/408821947 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9100b0a70 con 0x7fa9100b4d60 2024-03-25T17:59:51.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.494+0000 7fa90e7fc640 1 -- 172.21.15.18:0/408821947 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515476053 0 0) 0x7fa9100af940 con 0x7fa9100b1730 2024-03-25T17:59:51.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.494+0000 7fa90e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:59:51.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.494+0000 7fa90e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:51.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.494+0000 7fa90e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:51.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.494+0000 7fa90e7fc640 10 monclient(hunting): my global_id is 4380 2024-03-25T17:59:51.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.494+0000 7fa90e7fc640 1 -- 172.21.15.18:0/408821947 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9100b0d50 con 0x7fa9100b1730 2024-03-25T17:59:51.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.509+0000 7fa90e7fc640 1 -- 172.21.15.18:0/408821947 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4157403991 0 0) 0x7fa8f8003520 con 0x7fa91014be90 2024-03-25T17:59:51.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.509+0000 7fa90e7fc640 1 -- 172.21.15.18:0/408821947 >> v1:172.21.15.18:6790/0 conn(0x7fa9100b4d60 legacy=0x7fa91013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:51.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.509+0000 7fa90e7fc640 1 -- 172.21.15.18:0/408821947 >> v1:172.21.15.18:6789/0 conn(0x7fa9100b1730 legacy=0x7fa9100b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:51.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.509+0000 7fa90e7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:59:51.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.509+0000 7fa90e7fc640 1 monclient: found mon.noname-c 2024-03-25T17:59:51.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.509+0000 7fa90e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:59:51.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.509+0000 7fa90e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T17:59:51.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.509+0000 7fa90e7fc640 1 -- 172.21.15.18:0/408821947 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9100b1030 con 0x7fa91014be90 2024-03-25T17:59:51.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.509+0000 7fa90e7fc640 10 monclient: _finish_auth 0 2024-03-25T17:59:51.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.509+0000 7fa90e7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:59:51.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.509+0000 7fa90e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:59:51.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.509+0000 7fa90e7fc640 1 -- 172.21.15.18:0/408821947 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa908001a10 con 0x7fa91014be90 2024-03-25T17:59:51.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.509+0000 7fa90e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:51.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.509+0000 7fa90e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:59:51.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.509+0000 7fa90e7fc640 10 monclient: dump: 2024-03-25T17:59:51.510 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:51.511 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:51.511 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:51.511 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:51.511 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:51.511 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:51.511 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:51.511 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:51.511 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:51.511 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:51.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.509+0000 7fa9160d3640 5 monclient: authenticate success, global_id 4369 2024-03-25T17:59:51.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.509+0000 7fa9160d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:59:51.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.526+0000 7fa90e7fc640 1 -- 172.21.15.18:0/408821947 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa908001b70 con 0x7fa91014be90 2024-03-25T17:59:51.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.526+0000 7fa90e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:59:51.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.526+0000 7fa90e7fc640 1 -- 172.21.15.18:0/408821947 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa908001f80 con 0x7fa91014be90 2024-03-25T17:59:51.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.526+0000 7fa9160d3640 10 monclient: get_monmap_and_config success 2024-03-25T17:59:51.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.526+0000 7fa9160d3640 4 set_mon_vals no callback set 2024-03-25T17:59:51.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.526+0000 7fa90e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:51.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.526+0000 7fa90e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:59:51.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.526+0000 7fa90e7fc640 10 monclient: dump: 2024-03-25T17:59:51.526 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:51.526 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:51.526 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:51.526 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:51.526 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:51.526 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:51.526 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:51.527 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:51.527 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:51.527 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:51.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.526+0000 7fa9160d3640 10 monclient: shutdown 2024-03-25T17:59:51.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.526+0000 7fa9160d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:59:51.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.526+0000 7fa9160d3640 1 -- 172.21.15.18:0/408821947 >> v1:172.21.15.169:6789/0 conn(0x7fa91014be90 legacy=0x7fa91014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:51.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.527+0000 7fa9160d3640 1 -- 172.21.15.18:0/408821947 shutdown_connections 2024-03-25T17:59:51.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.527+0000 7fa9160d3640 1 -- 172.21.15.18:0/408821947 >> 172.21.15.18:0/408821947 conn(0x7fa9100617a0 msgr2=0x7fa910061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:59:51.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.527+0000 7fa9160d3640 1 -- 172.21.15.18:0/408821947 shutdown_connections 2024-03-25T17:59:51.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.527+0000 7fa9160d3640 1 -- 172.21.15.18:0/408821947 wait complete. 2024-03-25T17:59:51.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.527+0000 7fa9160d3640 10 monclient: build_initial_monmap 2024-03-25T17:59:51.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.527+0000 7fa9160d3640 10 monclient: monmap: 2024-03-25T17:59:51.529 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:59:51.529 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:59:51.529 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:59:51.529 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:59:51.529 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:59:51.529 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:59:51.529 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:59:51.529 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:59:51.529 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:59:51.529 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:51.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.528+0000 7fa9160d3640 1 librados: starting msgr at 2024-03-25T17:59:51.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.528+0000 7fa9160d3640 1 librados: starting objecter 2024-03-25T17:59:51.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.528+0000 7fa9160d3640 1 Processor -- start 2024-03-25T17:59:51.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.528+0000 7fa9160d3640 1 -- start start 2024-03-25T17:59:51.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.528+0000 7fa9160d3640 1 librados: setting wanted keys 2024-03-25T17:59:51.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.528+0000 7fa9160d3640 1 librados: calling monclient init 2024-03-25T17:59:51.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.528+0000 7fa9160d3640 10 monclient: init 2024-03-25T17:59:51.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.529+0000 7fa9160d3640 10 monclient: _reopen_session rank -1 2024-03-25T17:59:51.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.529+0000 7fa9160d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:59:51.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.529+0000 7fa9160d3640 10 monclient(hunting): picked mon.noname-c con 0x7fa9100b1730 addr v1:172.21.15.18:6790/0 2024-03-25T17:59:51.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.529+0000 7fa9160d3640 10 monclient(hunting): picked mon.noname-a con 0x7fa9100b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T17:59:51.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.529+0000 7fa9160d3640 10 monclient(hunting): picked mon.noname-b con 0x7fa91014be90 addr v1:172.21.15.169:6789/0 2024-03-25T17:59:51.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.529+0000 7fa9160d3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9101391b0 con 0x7fa9100b4d60 2024-03-25T17:59:51.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.529+0000 7fa9160d3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa910132750 con 0x7fa91014be90 2024-03-25T17:59:51.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.529+0000 7fa9160d3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa910138e40 con 0x7fa9100b1730 2024-03-25T17:59:51.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.529+0000 7fa9160d3640 10 monclient(hunting): _renew_subs 2024-03-25T17:59:51.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.529+0000 7fa9160d3640 10 monclient(hunting): authenticate will time out at 7120.107071s 2024-03-25T17:59:51.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.538+0000 7fa90ffff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fa91014be90 0x7fa9101db2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:32914/0 (socket says 172.21.15.18:32914) 2024-03-25T17:59:51.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.538+0000 7fa90ffff640 1 -- 172.21.15.18:0/3113215349 learned_addr learned my addr 172.21.15.18:0/3113215349 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:59:51.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.541+0000 7fa8f7fff640 10 client.?.objecter ms_handle_connect 0x7fa91014be90 2024-03-25T17:59:51.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.541+0000 7fa8f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:51.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.546+0000 7fa8f7fff640 10 client.?.objecter ms_handle_connect 0x7fa9100b1730 2024-03-25T17:59:51.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.546+0000 7fa8f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:51.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.547+0000 7fa8f7fff640 10 client.?.objecter ms_handle_connect 0x7fa9100b4d60 2024-03-25T17:59:51.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.547+0000 7fa8f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:51.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.548+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902029539 0 0) 0x7fa910138e40 con 0x7fa9100b1730 2024-03-25T17:59:51.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.548+0000 7fa8f7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:59:51.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.548+0000 7fa8f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:51.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.548+0000 7fa8f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:51.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.548+0000 7fa8f7fff640 10 monclient(hunting): my global_id is 4391 2024-03-25T17:59:51.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.548+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8fc004eb0 con 0x7fa9100b1730 2024-03-25T17:59:51.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.551+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021324321 0 0) 0x7fa9101391b0 con 0x7fa9100b4d60 2024-03-25T17:59:51.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.551+0000 7fa8f7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:59:51.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.551+0000 7fa8f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:51.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.551+0000 7fa8f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:51.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.551+0000 7fa8f7fff640 10 monclient(hunting): my global_id is 4383 2024-03-25T17:59:51.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.551+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa910138e40 con 0x7fa9100b4d60 2024-03-25T17:59:51.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.555+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1996010983 0 0) 0x7fa8fc004eb0 con 0x7fa9100b1730 2024-03-25T17:59:51.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.555+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9101391b0 con 0x7fa9100b1730 2024-03-25T17:59:51.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.555+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa900001970 con 0x7fa9100b1730 2024-03-25T17:59:51.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.555+0000 7fa8f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:51.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.555+0000 7fa8f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:59:51.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.555+0000 7fa8f7fff640 10 monclient(hunting): dump: 2024-03-25T17:59:51.555 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:51.555 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:51.555 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:51.555 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:51.556 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:51.556 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:51.556 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:51.556 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:51.556 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:51.556 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:51.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.555+0000 7fa8f7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:59:51.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.555+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902321485 0 0) 0x7fa910132750 con 0x7fa91014be90 2024-03-25T17:59:51.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.555+0000 7fa8f7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:59:51.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.555+0000 7fa8f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:51.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.555+0000 7fa8f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:51.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.555+0000 7fa8f7fff640 10 monclient(hunting): my global_id is 4372 2024-03-25T17:59:51.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.555+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8fc004eb0 con 0x7fa91014be90 2024-03-25T17:59:51.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 282659796 0 0) 0x7fa9101391b0 con 0x7fa9100b1730 2024-03-25T17:59:51.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 >> v1:172.21.15.169:6789/0 conn(0x7fa91014be90 legacy=0x7fa9101db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:51.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 >> v1:172.21.15.18:6789/0 conn(0x7fa9100b4d60 legacy=0x7fa9101dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:51.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa8f7fff640 10 monclient: _finish_hunting 0 2024-03-25T17:59:51.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa8f7fff640 1 monclient: found mon.c 2024-03-25T17:59:51.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa8f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:59:51.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa8f7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:59:51.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa910137c00 con 0x7fa9100b1730 2024-03-25T17:59:51.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa8f7fff640 10 monclient: _finish_auth 0 2024-03-25T17:59:51.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa8f7fff640 10 monclient: _check_auth_tickets 2024-03-25T17:59:51.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa8f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:59:51.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa9160d3640 5 monclient: authenticate success, global_id 4391 2024-03-25T17:59:51.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa9160d3640 10 monclient: _renew_subs 2024-03-25T17:59:51.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa9160d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:59:51.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa9160d3640 1 -- 172.21.15.18:0/3113215349 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa910138050 con 0x7fa9100b1730 2024-03-25T17:59:51.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa9160d3640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:59:51.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa9160d3640 10 monclient: _renew_subs 2024-03-25T17:59:51.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa9160d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:59:51.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa9160d3640 1 -- 172.21.15.18:0/3113215349 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa910145260 con 0x7fa9100b1730 2024-03-25T17:59:51.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.561+0000 7fa9160d3640 1 librados: init done 2024-03-25T17:59:51.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.562+0000 7fa9160d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:51.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.562+0000 7fa9160d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:51.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.562+0000 7fa9160d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:59:51.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.562+0000 7fa9160d3640 1 -- 172.21.15.18:0/3113215349 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9101401f0 con 0x7fa9100b1730 2024-03-25T17:59:51.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.873+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa900003610 con 0x7fa9100b1730 2024-03-25T17:59:51.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.873+0000 7fa8f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:59:51.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.873+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa900003920 con 0x7fa9100b1730 2024-03-25T17:59:51.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.873+0000 7fa8f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:51.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.873+0000 7fa8f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:59:51.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.873+0000 7fa8f7fff640 10 monclient: dump: 2024-03-25T17:59:51.874 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:51.874 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:51.874 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:51.874 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:51.875 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:51.875 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:51.875 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:51.875 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:51.875 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:51.875 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:51.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.873+0000 7fa90dffb640 4 set_mon_vals no callback set 2024-03-25T17:59:51.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.875+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fa9000198f0 con 0x7fa9100b1730 2024-03-25T17:59:51.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.875+0000 7fa90effd640 10 monclient: get_auth_request con 0x7fa8fc06de00 auth_method 0 2024-03-25T17:59:51.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.875+0000 7fa8f7fff640 10 client.4391.objecter ms_handle_connect 0x7fa8fc06de00 2024-03-25T17:59:51.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.880+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fa9000535f0 con 0x7fa9100b1730 2024-03-25T17:59:51.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.880+0000 7fa8f7fff640 10 client.4391.objecter ms_dispatch 0x7fa910000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:59:51.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.880+0000 7fa8f7fff640 3 client.4391.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:59:51.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.880+0000 7fa8f7fff640 3 client.4391.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:59:51.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.880+0000 7fa8f7fff640 20 client.4391.objecter dump_active .. 0 homeless 2024-03-25T17:59:51.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.880+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa90004ef30 con 0x7fa9100b1730 2024-03-25T17:59:51.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.880+0000 7fa8f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:51.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:51.880+0000 7fa8f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:59:52.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.125+0000 7fa9160d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:52.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.125+0000 7fa9160d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:52.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.125+0000 7fa9160d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T17:59:52.125 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.125+0000 7fa9160d3640 1 -- 172.21.15.18:0/3113215349 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa910139360 con 0x7fa9100b1730 2024-03-25T17:59:52.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.170+0000 7fa8f7fff640 1 -- 172.21.15.18:0/3113215349 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fa910139360 con 0x7fa9100b1730 2024-03-25T17:59:52.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.170+0000 7fa8f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:52.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.170+0000 7fa8f7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:59:52.171 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:59:52.171 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:59:52.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.173+0000 7fa9160d3640 10 librados: watch_flush enter 2024-03-25T17:59:52.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.173+0000 7fa9160d3640 10 librados: watch_flush exit 2024-03-25T17:59:52.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.173+0000 7fa9160d3640 20 client.4391.objecter shutdown clearing up homeless session... 2024-03-25T17:59:52.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.173+0000 7fa9160d3640 10 client.4391.objecter successfully canceled tick 2024-03-25T17:59:52.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.173+0000 7fa9160d3640 1 -- 172.21.15.18:0/3113215349 >> v1:172.21.15.18:6800/3721244451 conn(0x7fa8fc06de00 legacy=0x7fa8fc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:52.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.173+0000 7fa9160d3640 10 monclient: shutdown 2024-03-25T17:59:52.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.173+0000 7fa9160d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:59:52.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.173+0000 7fa9160d3640 1 -- 172.21.15.18:0/3113215349 >> v1:172.21.15.18:6790/0 conn(0x7fa9100b1730 legacy=0x7fa910132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:52.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.173+0000 7fa9160d3640 1 -- 172.21.15.18:0/3113215349 shutdown_connections 2024-03-25T17:59:52.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.173+0000 7fa9160d3640 1 -- 172.21.15.18:0/3113215349 >> 172.21.15.18:0/3113215349 conn(0x7fa9100617a0 msgr2=0x7fa9100b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:59:52.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.173+0000 7fa9160d3640 1 -- 172.21.15.18:0/3113215349 shutdown_connections 2024-03-25T17:59:52.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.173+0000 7fa9160d3640 1 -- 172.21.15.18:0/3113215349 wait complete. 2024-03-25T17:59:52.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:52.174+0000 7fa9160d3640 1 librados: shutdown 2024-03-25T17:59:52.185 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:59:58.992 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:59:59.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.107+0000 7f939630e640 10 monclient: get_monmap_and_config 2024-03-25T17:59:59.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.107+0000 7f939630e640 10 monclient: build_initial_monmap 2024-03-25T17:59:59.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.107+0000 7f939630e640 10 monclient: monmap: 2024-03-25T17:59:59.107 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:59:59.107 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:59.107 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:59:59.107674+0000 2024-03-25T17:59:59.107 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:59:59.107674+0000 2024-03-25T17:59:59.107 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:59:59.107 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:59.108 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:59:59.108 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T17:59:59.108 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T17:59:59.108 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:59.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.108+0000 7f939630e640 1 Processor -- start 2024-03-25T17:59:59.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.108+0000 7f939630e640 1 -- start start 2024-03-25T17:59:59.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.108+0000 7f939630e640 10 monclient: init 2024-03-25T17:59:59.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.108+0000 7f939630e640 10 monclient: _reopen_session rank -1 2024-03-25T17:59:59.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.108+0000 7f939630e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:59:59.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.108+0000 7f939630e640 10 monclient(hunting): picked mon.noname-a con 0x7f93900a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T17:59:59.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.108+0000 7f939630e640 10 monclient(hunting): picked mon.noname-c con 0x7f939013f710 addr v1:172.21.15.169:6789/0 2024-03-25T17:59:59.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.108+0000 7f939630e640 10 monclient(hunting): picked mon.noname-b con 0x7f93901430c0 addr v1:172.21.15.18:6790/0 2024-03-25T17:59:59.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.108+0000 7f939630e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9390142520 con 0x7f93900a98f0 2024-03-25T17:59:59.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.108+0000 7f939630e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93900b28a0 con 0x7f939013f710 2024-03-25T17:59:59.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.108+0000 7f939630e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93900b2a30 con 0x7f93901430c0 2024-03-25T17:59:59.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.108+0000 7f939630e640 10 monclient(hunting): _renew_subs 2024-03-25T17:59:59.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.108+0000 7f939630e640 10 monclient(hunting): authenticate will time out at 7127.686509s 2024-03-25T17:59:59.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.115+0000 7f938f7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f939013f710 0x7f93901429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:56232/0 (socket says 172.21.15.18:56232) 2024-03-25T17:59:59.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.115+0000 7f938f7fe640 1 -- 172.21.15.18:0/3158276691 learned_addr learned my addr 172.21.15.18:0/3158276691 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:59:59.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.127+0000 7f938effd640 1 -- 172.21.15.18:0/3158276691 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291179409 0 0) 0x7f9390142520 con 0x7f93900a98f0 2024-03-25T17:59:59.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.128+0000 7f938effd640 10 monclient(hunting): _negotiate 2024-03-25T17:59:59.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.128+0000 7f938effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:59.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.128+0000 7f938effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:59.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.128+0000 7f938effd640 10 monclient(hunting): my global_id is 4386 2024-03-25T17:59:59.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.128+0000 7f938effd640 1 -- 172.21.15.18:0/3158276691 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9378003520 con 0x7f93900a98f0 2024-03-25T17:59:59.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.134+0000 7f938effd640 1 -- 172.21.15.18:0/3158276691 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1881087444 0 0) 0x7f9378003520 con 0x7f93900a98f0 2024-03-25T17:59:59.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.134+0000 7f938effd640 1 -- 172.21.15.18:0/3158276691 >> v1:172.21.15.18:6790/0 conn(0x7f93901430c0 legacy=0x7f939014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:59.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.134+0000 7f938effd640 1 -- 172.21.15.18:0/3158276691 >> v1:172.21.15.169:6789/0 conn(0x7f939013f710 legacy=0x7f93901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:59.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.134+0000 7f938effd640 10 monclient: _finish_hunting 0 2024-03-25T17:59:59.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.134+0000 7f938effd640 1 monclient: found mon.noname-a 2024-03-25T17:59:59.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.134+0000 7f938effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:59:59.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.134+0000 7f938effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T17:59:59.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.134+0000 7f938effd640 1 -- 172.21.15.18:0/3158276691 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93900b44c0 con 0x7f93900a98f0 2024-03-25T17:59:59.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.134+0000 7f938effd640 10 monclient: _finish_auth 0 2024-03-25T17:59:59.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.134+0000 7f938effd640 10 monclient: _check_auth_tickets 2024-03-25T17:59:59.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.134+0000 7f938effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:59:59.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.134+0000 7f939630e640 5 monclient: authenticate success, global_id 4386 2024-03-25T17:59:59.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.134+0000 7f939630e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:59:59.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.135+0000 7f938effd640 1 -- 172.21.15.18:0/3158276691 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9380001690 con 0x7f93900a98f0 2024-03-25T17:59:59.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.135+0000 7f938effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:59.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.135+0000 7f938effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:59:59.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.135+0000 7f938effd640 10 monclient: dump: 2024-03-25T17:59:59.135 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:59.136 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:59.136 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:59.136 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:59.136 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:59.136 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:59.136 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:59.136 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:59.136 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:59.136 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:59.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.135+0000 7f939630e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:59:59.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.154+0000 7f938effd640 1 -- 172.21.15.18:0/3158276691 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9380001920 con 0x7f93900a98f0 2024-03-25T17:59:59.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.154+0000 7f938effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:59:59.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.154+0000 7f938effd640 1 -- 172.21.15.18:0/3158276691 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9380001c30 con 0x7f93900a98f0 2024-03-25T17:59:59.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.154+0000 7f939630e640 10 monclient: get_monmap_and_config success 2024-03-25T17:59:59.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.154+0000 7f939630e640 4 set_mon_vals no callback set 2024-03-25T17:59:59.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.154+0000 7f938effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:59.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.154+0000 7f938effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:59:59.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.154+0000 7f938effd640 10 monclient: dump: 2024-03-25T17:59:59.154 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:59.155 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:59.155 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:59.155 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:59.155 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:59.155 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:59.155 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:59.155 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:59.155 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:59.155 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:59.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.155+0000 7f939630e640 10 monclient: shutdown 2024-03-25T17:59:59.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.155+0000 7f939630e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:59:59.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.155+0000 7f939630e640 1 -- 172.21.15.18:0/3158276691 >> v1:172.21.15.18:6789/0 conn(0x7f93900a98f0 legacy=0x7f93900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:59.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.155+0000 7f939630e640 1 -- 172.21.15.18:0/3158276691 shutdown_connections 2024-03-25T17:59:59.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.155+0000 7f939630e640 1 -- 172.21.15.18:0/3158276691 >> 172.21.15.18:0/3158276691 conn(0x7f93900b7ea0 msgr2=0x7f93900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:59:59.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.155+0000 7f939630e640 1 -- 172.21.15.18:0/3158276691 shutdown_connections 2024-03-25T17:59:59.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.155+0000 7f939630e640 1 -- 172.21.15.18:0/3158276691 wait complete. 2024-03-25T17:59:59.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.155+0000 7f939630e640 10 monclient: build_initial_monmap 2024-03-25T17:59:59.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.155+0000 7f939630e640 10 monclient: monmap: 2024-03-25T17:59:59.158 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T17:59:59.158 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:59:59.158 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T17:59:59.158 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T17:59:59.158 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T17:59:59.158 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T17:59:59.158 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T17:59:59.158 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T17:59:59.158 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T17:59:59.158 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:59.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.156+0000 7f939630e640 1 librados: starting msgr at 2024-03-25T17:59:59.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.156+0000 7f939630e640 1 librados: starting objecter 2024-03-25T17:59:59.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.156+0000 7f939630e640 1 Processor -- start 2024-03-25T17:59:59.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.156+0000 7f939630e640 1 -- start start 2024-03-25T17:59:59.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.156+0000 7f939630e640 1 librados: setting wanted keys 2024-03-25T17:59:59.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.156+0000 7f939630e640 1 librados: calling monclient init 2024-03-25T17:59:59.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.156+0000 7f939630e640 10 monclient: init 2024-03-25T17:59:59.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.157+0000 7f939630e640 10 monclient: _reopen_session rank -1 2024-03-25T17:59:59.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.157+0000 7f939630e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:59:59.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.157+0000 7f939630e640 10 monclient(hunting): picked mon.noname-c con 0x7f93900a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T17:59:59.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.157+0000 7f939630e640 10 monclient(hunting): picked mon.noname-a con 0x7f939013f710 addr v1:172.21.15.18:6789/0 2024-03-25T17:59:59.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.157+0000 7f939630e640 10 monclient(hunting): picked mon.noname-b con 0x7f93901430c0 addr v1:172.21.15.169:6789/0 2024-03-25T17:59:59.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.157+0000 7f939630e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9390141c10 con 0x7f939013f710 2024-03-25T17:59:59.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.157+0000 7f939630e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9390135fc0 con 0x7f93901430c0 2024-03-25T17:59:59.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.157+0000 7f939630e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93901417c0 con 0x7f93900a98f0 2024-03-25T17:59:59.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.157+0000 7f939630e640 10 monclient(hunting): _renew_subs 2024-03-25T17:59:59.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.157+0000 7f939630e640 10 monclient(hunting): authenticate will time out at 7127.734834s 2024-03-25T17:59:59.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.157+0000 7f938ffff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f93900a98f0 0x7f93901daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:41872/0 (socket says 172.21.15.18:41872) 2024-03-25T17:59:59.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.157+0000 7f938ffff640 1 -- 172.21.15.18:0/2374352169 learned_addr learned my addr 172.21.15.18:0/2374352169 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T17:59:59.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.158+0000 7f938cff9640 10 client.?.objecter ms_handle_connect 0x7f93900a98f0 2024-03-25T17:59:59.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.158+0000 7f938cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:59.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.158+0000 7f938cff9640 10 client.?.objecter ms_handle_connect 0x7f93901430c0 2024-03-25T17:59:59.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.158+0000 7f938cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:59.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.232+0000 7f938cff9640 10 client.?.objecter ms_handle_connect 0x7f939013f710 2024-03-25T17:59:59.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.232+0000 7f938cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:59:59.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.253+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471785525 0 0) 0x7f9390141c10 con 0x7f939013f710 2024-03-25T17:59:59.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.253+0000 7f938cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:59:59.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.253+0000 7f938cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:59.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.253+0000 7f938cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:59.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.253+0000 7f938cff9640 10 monclient(hunting): my global_id is 4389 2024-03-25T17:59:59.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.253+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f937c004eb0 con 0x7f939013f710 2024-03-25T17:59:59.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.263+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263184750 0 0) 0x7f9390135fc0 con 0x7f93901430c0 2024-03-25T17:59:59.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.263+0000 7f938cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:59:59.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.263+0000 7f938cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:59.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.263+0000 7f938cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:59.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.263+0000 7f938cff9640 10 monclient(hunting): my global_id is 4375 2024-03-25T17:59:59.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.263+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9390141c10 con 0x7f93901430c0 2024-03-25T17:59:59.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.270+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1827313300 0 0) 0x7f937c004eb0 con 0x7f939013f710 2024-03-25T17:59:59.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.270+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9390135fc0 con 0x7f939013f710 2024-03-25T17:59:59.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.270+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f93840018a0 con 0x7f939013f710 2024-03-25T17:59:59.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.270+0000 7f938cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:59.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.270+0000 7f938cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:59:59.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.270+0000 7f938cff9640 10 monclient(hunting): dump: 2024-03-25T17:59:59.270 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:59.270 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:59.270 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:59.270 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:59.270 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:59.270 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:59.270 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:59.271 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:59.271 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:59.271 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:59.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.270+0000 7f938cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:59:59.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.281+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695879372 0 0) 0x7f93901417c0 con 0x7f93900a98f0 2024-03-25T17:59:59.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.281+0000 7f938cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:59:59.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.281+0000 7f938cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:59:59.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.281+0000 7f938cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:59:59.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.281+0000 7f938cff9640 10 monclient(hunting): my global_id is 4394 2024-03-25T17:59:59.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.281+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f937c004eb0 con 0x7f93900a98f0 2024-03-25T17:59:59.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4175768050 0 0) 0x7f9390135fc0 con 0x7f939013f710 2024-03-25T17:59:59.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 >> v1:172.21.15.18:6790/0 conn(0x7f93900a98f0 legacy=0x7f93901daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:59.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 >> v1:172.21.15.169:6789/0 conn(0x7f93901430c0 legacy=0x7f93901e1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:59.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f938cff9640 10 monclient: _finish_hunting 0 2024-03-25T17:59:59.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f938cff9640 1 monclient: found mon.a 2024-03-25T17:59:59.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f938cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:59:59.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f938cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:59.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9390141950 con 0x7f939013f710 2024-03-25T17:59:59.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f938cff9640 10 monclient: _finish_auth 0 2024-03-25T17:59:59.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f938cff9640 10 monclient: _check_auth_tickets 2024-03-25T17:59:59.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f938cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:59:59.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f939630e640 5 monclient: authenticate success, global_id 4389 2024-03-25T17:59:59.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f939630e640 10 monclient: _renew_subs 2024-03-25T17:59:59.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f939630e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:59.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f939630e640 1 -- 172.21.15.18:0/2374352169 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93901353b0 con 0x7f939013f710 2024-03-25T17:59:59.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f939630e640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:59:59.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f939630e640 10 monclient: _renew_subs 2024-03-25T17:59:59.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f939630e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:59.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f939630e640 1 -- 172.21.15.18:0/2374352169 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93900b7900 con 0x7f939013f710 2024-03-25T17:59:59.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.286+0000 7f939630e640 1 librados: init done 2024-03-25T17:59:59.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.287+0000 7f939630e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:59.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.287+0000 7f939630e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:59.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.287+0000 7f939630e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:59.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.287+0000 7f939630e640 1 -- 172.21.15.18:0/2374352169 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93900627b0 con 0x7f939013f710 2024-03-25T17:59:59.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.508+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9384001d10 con 0x7f939013f710 2024-03-25T17:59:59.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.508+0000 7f938cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:59:59.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.508+0000 7f938e7fc640 4 set_mon_vals no callback set 2024-03-25T17:59:59.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.558+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9384004980 con 0x7f939013f710 2024-03-25T17:59:59.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.558+0000 7f938cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:59:59.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.558+0000 7f938cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:59:59.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.558+0000 7f938cff9640 10 monclient: dump: 2024-03-25T17:59:59.558 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T17:59:59.558 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T17:59:59.558 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:59.558 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T17:59:59.558 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T17:59:59.558 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T17:59:59.559 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T17:59:59.559 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T17:59:59.559 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T17:59:59.559 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T17:59:59.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.558+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f9384004b60 con 0x7f939013f710 2024-03-25T17:59:59.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.559+0000 7f938ffff640 10 monclient: get_auth_request con 0x7f937c06dda0 auth_method 0 2024-03-25T17:59:59.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.559+0000 7f938cff9640 10 client.4389.objecter ms_handle_connect 0x7f937c06dda0 2024-03-25T17:59:59.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.574+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f93840547a0 con 0x7f939013f710 2024-03-25T17:59:59.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.574+0000 7f938cff9640 10 client.4389.objecter ms_dispatch 0x7f9390000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:59:59.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.574+0000 7f938cff9640 3 client.4389.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:59:59.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.574+0000 7f938cff9640 3 client.4389.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:59:59.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.574+0000 7f938cff9640 20 client.4389.objecter dump_active .. 0 homeless 2024-03-25T17:59:59.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.574+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f93840500e0 con 0x7f939013f710 2024-03-25T17:59:59.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.574+0000 7f938cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:59:59.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.574+0000 7f938cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:59:59.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.797+0000 7f939630e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:59.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.797+0000 7f939630e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:59.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.797+0000 7f939630e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T17:59:59.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.797+0000 7f939630e640 1 -- 172.21.15.18:0/2374352169 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9390141da0 con 0x7f939013f710 2024-03-25T17:59:59.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.824+0000 7f938cff9640 1 -- 172.21.15.18:0/2374352169 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f93840521e0 con 0x7f939013f710 2024-03-25T17:59:59.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.824+0000 7f938cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:59:59.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.824+0000 7f938cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:59:59.824 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T17:59:59.824 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:59:59.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.827+0000 7f939630e640 10 librados: watch_flush enter 2024-03-25T17:59:59.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.827+0000 7f939630e640 10 librados: watch_flush exit 2024-03-25T17:59:59.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.827+0000 7f939630e640 20 client.4389.objecter shutdown clearing up homeless session... 2024-03-25T17:59:59.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.827+0000 7f939630e640 10 client.4389.objecter successfully canceled tick 2024-03-25T17:59:59.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.827+0000 7f939630e640 1 -- 172.21.15.18:0/2374352169 >> v1:172.21.15.18:6800/3721244451 conn(0x7f937c06dda0 legacy=0x7f937c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:59.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.827+0000 7f939630e640 10 monclient: shutdown 2024-03-25T17:59:59.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.827+0000 7f939630e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:59:59.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.827+0000 7f939630e640 1 -- 172.21.15.18:0/2374352169 >> v1:172.21.15.18:6789/0 conn(0x7f939013f710 legacy=0x7f93901db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:59:59.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.827+0000 7f939630e640 1 -- 172.21.15.18:0/2374352169 shutdown_connections 2024-03-25T17:59:59.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.827+0000 7f939630e640 1 -- 172.21.15.18:0/2374352169 >> 172.21.15.18:0/2374352169 conn(0x7f93900b7ea0 msgr2=0x7f9390074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:59:59.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.827+0000 7f939630e640 1 -- 172.21.15.18:0/2374352169 shutdown_connections 2024-03-25T17:59:59.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.827+0000 7f939630e640 1 -- 172.21.15.18:0/2374352169 wait complete. 2024-03-25T17:59:59.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T17:59:59.827+0000 7f939630e640 1 librados: shutdown 2024-03-25T17:59:59.839 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:00:02.689 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:04:00.482 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:02.695 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:04:00.488 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:02.729 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:04:00.522 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:02.744 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:04:00.493 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:02.750 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:04:00.499 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:02.776 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:04:00.569 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:02.778 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:04:00.528 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:02.789 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:04:00.582 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:02.827 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:04:00.620 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:06.645 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:00:06.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.762+0000 7f7c5524a640 10 monclient: get_monmap_and_config 2024-03-25T18:00:06.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.762+0000 7f7c5524a640 10 monclient: build_initial_monmap 2024-03-25T18:00:06.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.762+0000 7f7c5524a640 10 monclient: monmap: 2024-03-25T18:00:06.762 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:00:06.762 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:06.762 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:00:06.762415+0000 2024-03-25T18:00:06.762 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:00:06.762415+0000 2024-03-25T18:00:06.762 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:00:06.762 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:06.762 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:00:06.763 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:00:06.763 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:00:06.763 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:06.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.762+0000 7f7c5524a640 1 Processor -- start 2024-03-25T18:00:06.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.763+0000 7f7c5524a640 1 -- start start 2024-03-25T18:00:06.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.763+0000 7f7c5524a640 10 monclient: init 2024-03-25T18:00:06.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.763+0000 7f7c5524a640 10 monclient: _reopen_session rank -1 2024-03-25T18:00:06.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.763+0000 7f7c5524a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:00:06.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.763+0000 7f7c5524a640 10 monclient(hunting): picked mon.noname-a con 0x7f7c500a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:00:06.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.763+0000 7f7c5524a640 10 monclient(hunting): picked mon.noname-c con 0x7f7c5013f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:00:06.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.763+0000 7f7c5524a640 10 monclient(hunting): picked mon.noname-b con 0x7f7c501430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:00:06.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.763+0000 7f7c5524a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c50142520 con 0x7f7c500a98f0 2024-03-25T18:00:06.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.763+0000 7f7c5524a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c500b28a0 con 0x7f7c5013f710 2024-03-25T18:00:06.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.763+0000 7f7c5524a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c500b2a30 con 0x7f7c501430c0 2024-03-25T18:00:06.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.763+0000 7f7c5524a640 10 monclient(hunting): _renew_subs 2024-03-25T18:00:06.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.763+0000 7f7c5524a640 10 monclient(hunting): authenticate will time out at 7135.341486s 2024-03-25T18:00:06.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.767+0000 7f7c4ed9d640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f7c500a98f0 0x7f7c500a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:48496/0 (socket says 172.21.15.18:48496) 2024-03-25T18:00:06.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.767+0000 7f7c4ed9d640 1 -- 172.21.15.18:0/3884354315 learned_addr learned my addr 172.21.15.18:0/3884354315 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:00:06.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.790+0000 7f7c4dd9b640 1 -- 172.21.15.18:0/3884354315 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435337012 0 0) 0x7f7c50142520 con 0x7f7c500a98f0 2024-03-25T18:00:06.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.790+0000 7f7c4dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:00:06.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.790+0000 7f7c4dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:06.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.790+0000 7f7c4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:06.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.790+0000 7f7c4dd9b640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:00:06.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.790+0000 7f7c4dd9b640 1 -- 172.21.15.18:0/3884354315 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c38003520 con 0x7f7c500a98f0 2024-03-25T18:00:06.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.799+0000 7f7c4dd9b640 1 -- 172.21.15.18:0/3884354315 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586081095 0 0) 0x7f7c500b28a0 con 0x7f7c5013f710 2024-03-25T18:00:06.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.799+0000 7f7c4dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:00:06.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.799+0000 7f7c4dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:06.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.799+0000 7f7c4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:06.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.799+0000 7f7c4dd9b640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:00:06.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.799+0000 7f7c4dd9b640 1 -- 172.21.15.18:0/3884354315 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c50142520 con 0x7f7c5013f710 2024-03-25T18:00:06.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.799+0000 7f7c4dd9b640 1 -- 172.21.15.18:0/3884354315 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3576436452 0 0) 0x7f7c38003520 con 0x7f7c500a98f0 2024-03-25T18:00:06.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.800+0000 7f7c4dd9b640 1 -- 172.21.15.18:0/3884354315 >> v1:172.21.15.18:6790/0 conn(0x7f7c501430c0 legacy=0x7f7c5014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:06.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.800+0000 7f7c4dd9b640 1 -- 172.21.15.18:0/3884354315 >> v1:172.21.15.169:6789/0 conn(0x7f7c5013f710 legacy=0x7f7c501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:06.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.800+0000 7f7c4dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:00:06.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.800+0000 7f7c4dd9b640 1 monclient: found mon.noname-a 2024-03-25T18:00:06.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.800+0000 7f7c4dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:00:06.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.800+0000 7f7c4dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:00:06.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.800+0000 7f7c4dd9b640 1 -- 172.21.15.18:0/3884354315 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c500b44c0 con 0x7f7c500a98f0 2024-03-25T18:00:06.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.800+0000 7f7c4dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:00:06.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.800+0000 7f7c4dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:00:06.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.800+0000 7f7c4dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:00:06.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.800+0000 7f7c5524a640 5 monclient: authenticate success, global_id 4392 2024-03-25T18:00:06.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.800+0000 7f7c5524a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:00:06.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.800+0000 7f7c4dd9b640 1 -- 172.21.15.18:0/3884354315 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f7c40001550 con 0x7f7c500a98f0 2024-03-25T18:00:06.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.800+0000 7f7c4dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:06.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.800+0000 7f7c4dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:00:06.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.800+0000 7f7c4dd9b640 10 monclient: dump: 2024-03-25T18:00:06.801 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:06.801 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:06.801 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:06.802 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:06.802 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:06.802 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:06.802 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:06.802 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:06.802 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:06.802 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:06.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.800+0000 7f7c5524a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:00:06.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.817+0000 7f7c4dd9b640 1 -- 172.21.15.18:0/3884354315 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c40001820 con 0x7f7c500a98f0 2024-03-25T18:00:06.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.817+0000 7f7c4dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:00:06.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.817+0000 7f7c4dd9b640 1 -- 172.21.15.18:0/3884354315 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f7c40001bb0 con 0x7f7c500a98f0 2024-03-25T18:00:06.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.817+0000 7f7c4dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:06.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.817+0000 7f7c4dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:00:06.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.817+0000 7f7c4dd9b640 10 monclient: dump: 2024-03-25T18:00:06.817 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:06.817 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:06.817 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:06.817 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:06.817 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:06.818 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:06.818 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:06.818 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:06.818 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:06.818 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:06.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.817+0000 7f7c5524a640 10 monclient: get_monmap_and_config success 2024-03-25T18:00:06.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.817+0000 7f7c5524a640 4 set_mon_vals no callback set 2024-03-25T18:00:06.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.818+0000 7f7c5524a640 10 monclient: shutdown 2024-03-25T18:00:06.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.818+0000 7f7c5524a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:00:06.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.818+0000 7f7c5524a640 1 -- 172.21.15.18:0/3884354315 >> v1:172.21.15.18:6789/0 conn(0x7f7c500a98f0 legacy=0x7f7c500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:06.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.818+0000 7f7c5524a640 1 -- 172.21.15.18:0/3884354315 shutdown_connections 2024-03-25T18:00:06.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.818+0000 7f7c5524a640 1 -- 172.21.15.18:0/3884354315 >> 172.21.15.18:0/3884354315 conn(0x7f7c500b7ea0 msgr2=0x7f7c500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:00:06.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.818+0000 7f7c5524a640 1 -- 172.21.15.18:0/3884354315 shutdown_connections 2024-03-25T18:00:06.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.818+0000 7f7c5524a640 1 -- 172.21.15.18:0/3884354315 wait complete. 2024-03-25T18:00:06.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.819+0000 7f7c5524a640 10 monclient: build_initial_monmap 2024-03-25T18:00:06.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.819+0000 7f7c5524a640 10 monclient: monmap: 2024-03-25T18:00:06.820 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:00:06.820 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:00:06.820 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:00:06.820 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:00:06.820 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:00:06.820 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:00:06.820 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:00:06.820 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:00:06.820 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:00:06.820 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:06.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.819+0000 7f7c5524a640 1 librados: starting msgr at 2024-03-25T18:00:06.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.819+0000 7f7c5524a640 1 librados: starting objecter 2024-03-25T18:00:06.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.820+0000 7f7c5524a640 1 Processor -- start 2024-03-25T18:00:06.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.820+0000 7f7c5524a640 1 -- start start 2024-03-25T18:00:06.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.820+0000 7f7c5524a640 1 librados: setting wanted keys 2024-03-25T18:00:06.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.820+0000 7f7c5524a640 1 librados: calling monclient init 2024-03-25T18:00:06.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.820+0000 7f7c5524a640 10 monclient: init 2024-03-25T18:00:06.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.820+0000 7f7c5524a640 10 monclient: _reopen_session rank -1 2024-03-25T18:00:06.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.820+0000 7f7c5524a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:00:06.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.820+0000 7f7c5524a640 10 monclient(hunting): picked mon.noname-a con 0x7f7c500a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:00:06.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.821+0000 7f7c5524a640 10 monclient(hunting): picked mon.noname-b con 0x7f7c5013f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:00:06.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.821+0000 7f7c5524a640 10 monclient(hunting): picked mon.noname-c con 0x7f7c501430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:00:06.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.821+0000 7f7c5524a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c50135280 con 0x7f7c500a98f0 2024-03-25T18:00:06.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.821+0000 7f7c5524a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c50136040 con 0x7f7c5013f710 2024-03-25T18:00:06.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.821+0000 7f7c5524a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c50131660 con 0x7f7c501430c0 2024-03-25T18:00:06.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.821+0000 7f7c5524a640 10 monclient(hunting): _renew_subs 2024-03-25T18:00:06.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.821+0000 7f7c5524a640 10 monclient(hunting): authenticate will time out at 7135.398753s 2024-03-25T18:00:06.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.823+0000 7f7c4e59c640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f7c5013f710 0x7f7c50130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:50340/0 (socket says 172.21.15.18:50340) 2024-03-25T18:00:06.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.823+0000 7f7c4e59c640 1 -- 172.21.15.18:0/2992698960 learned_addr learned my addr 172.21.15.18:0/2992698960 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:00:06.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.825+0000 7f7c377fe640 10 client.?.objecter ms_handle_connect 0x7f7c5013f710 2024-03-25T18:00:06.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.825+0000 7f7c377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:06.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.831+0000 7f7c377fe640 1 -- 172.21.15.18:0/2992698960 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394140949 0 0) 0x7f7c50136040 con 0x7f7c5013f710 2024-03-25T18:00:06.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.831+0000 7f7c377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:00:06.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.831+0000 7f7c377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:06.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.831+0000 7f7c377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:06.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.831+0000 7f7c377fe640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:00:06.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.831+0000 7f7c377fe640 1 -- 172.21.15.18:0/2992698960 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c3c004eb0 con 0x7f7c5013f710 2024-03-25T18:00:06.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.841+0000 7f7c377fe640 1 -- 172.21.15.18:0/2992698960 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 819060008 0 0) 0x7f7c3c004eb0 con 0x7f7c5013f710 2024-03-25T18:00:06.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.841+0000 7f7c377fe640 1 -- 172.21.15.18:0/2992698960 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c50136040 con 0x7f7c5013f710 2024-03-25T18:00:06.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.841+0000 7f7c377fe640 1 -- 172.21.15.18:0/2992698960 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f7c44001a10 con 0x7f7c5013f710 2024-03-25T18:00:06.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.841+0000 7f7c377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:06.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.841+0000 7f7c377fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:00:06.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.841+0000 7f7c377fe640 10 monclient(hunting): dump: 2024-03-25T18:00:06.841 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:06.841 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:06.841 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:06.841 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:06.841 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:06.841 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:06.841 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:06.841 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:06.841 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:06.842 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:06.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.841+0000 7f7c377fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:00:06.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.842+0000 7f7c377fe640 10 client.?.objecter ms_handle_connect 0x7f7c501430c0 2024-03-25T18:00:06.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.842+0000 7f7c377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:06.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c377fe640 1 -- 172.21.15.18:0/2992698960 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1414909779 0 0) 0x7f7c50136040 con 0x7f7c5013f710 2024-03-25T18:00:06.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c377fe640 1 -- 172.21.15.18:0/2992698960 >> v1:172.21.15.18:6790/0 conn(0x7f7c501430c0 legacy=0x7f7c501d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:06.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c377fe640 1 -- 172.21.15.18:0/2992698960 >> v1:172.21.15.18:6789/0 conn(0x7f7c500a98f0 legacy=0x7f7c50135930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:06.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c377fe640 10 monclient: _finish_hunting 0 2024-03-25T18:00:06.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c377fe640 1 monclient: found mon.b 2024-03-25T18:00:06.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:00:06.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c377fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:06.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c377fe640 1 -- 172.21.15.18:0/2992698960 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c501317f0 con 0x7f7c5013f710 2024-03-25T18:00:06.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c377fe640 10 monclient: _finish_auth 0 2024-03-25T18:00:06.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c377fe640 10 monclient: _check_auth_tickets 2024-03-25T18:00:06.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:00:06.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c5524a640 5 monclient: authenticate success, global_id 4381 2024-03-25T18:00:06.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c5524a640 10 monclient: _renew_subs 2024-03-25T18:00:06.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c5524a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:06.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c5524a640 1 -- 172.21.15.18:0/2992698960 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c5013b780 con 0x7f7c5013f710 2024-03-25T18:00:06.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c5524a640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:00:06.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c5524a640 10 monclient: _renew_subs 2024-03-25T18:00:06.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c5524a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:06.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c5524a640 1 -- 172.21.15.18:0/2992698960 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c501d8040 con 0x7f7c5013f710 2024-03-25T18:00:06.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.846+0000 7f7c5524a640 1 librados: init done 2024-03-25T18:00:06.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.847+0000 7f7c5524a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:06.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.847+0000 7f7c5524a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:06.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.847+0000 7f7c5524a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:06.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:06.847+0000 7f7c5524a640 1 -- 172.21.15.18:0/2992698960 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c5013d010 con 0x7f7c5013f710 2024-03-25T18:00:07.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.119+0000 7f7c377fe640 1 -- 172.21.15.18:0/2992698960 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c44001d10 con 0x7f7c5013f710 2024-03-25T18:00:07.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.119+0000 7f7c377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:00:07.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.119+0000 7f7c377fe640 1 -- 172.21.15.18:0/2992698960 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f7c44003a60 con 0x7f7c5013f710 2024-03-25T18:00:07.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.119+0000 7f7c377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:07.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.119+0000 7f7c377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:00:07.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.119+0000 7f7c377fe640 10 monclient: dump: 2024-03-25T18:00:07.121 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:07.121 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:07.121 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:07.121 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:07.121 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:07.121 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:07.121 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:07.121 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:07.121 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:07.121 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:07.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.119+0000 7f7c4d59a640 4 set_mon_vals no callback set 2024-03-25T18:00:07.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.120+0000 7f7c377fe640 1 -- 172.21.15.18:0/2992698960 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f7c44003c80 con 0x7f7c5013f710 2024-03-25T18:00:07.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.121+0000 7f7c4ed9d640 10 monclient: get_auth_request con 0x7f7c3c06dbf0 auth_method 0 2024-03-25T18:00:07.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.121+0000 7f7c377fe640 10 client.4381.objecter ms_handle_connect 0x7f7c3c06dbf0 2024-03-25T18:00:07.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.126+0000 7f7c377fe640 1 -- 172.21.15.18:0/2992698960 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f7c44054950 con 0x7f7c5013f710 2024-03-25T18:00:07.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.126+0000 7f7c377fe640 10 client.4381.objecter ms_dispatch 0x7f7c50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:00:07.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.126+0000 7f7c377fe640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:00:07.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.126+0000 7f7c377fe640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:00:07.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.126+0000 7f7c377fe640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-25T18:00:07.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.127+0000 7f7c377fe640 1 -- 172.21.15.18:0/2992698960 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c44050460 con 0x7f7c5013f710 2024-03-25T18:00:07.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.127+0000 7f7c377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:07.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.127+0000 7f7c377fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:00:07.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.359+0000 7f7c5524a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:07.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.359+0000 7f7c5524a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:07.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.359+0000 7f7c5524a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:07.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.359+0000 7f7c5524a640 1 -- 172.21.15.18:0/2992698960 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c501d8340 con 0x7f7c5013f710 2024-03-25T18:00:07.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.500+0000 7f7c377fe640 1 -- 172.21.15.18:0/2992698960 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f7c44052390 con 0x7f7c5013f710 2024-03-25T18:00:07.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.500+0000 7f7c377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:07.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.500+0000 7f7c377fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:00:07.501 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:00:07.501 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:00:07.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.502+0000 7f7c5524a640 10 librados: watch_flush enter 2024-03-25T18:00:07.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.502+0000 7f7c5524a640 10 librados: watch_flush exit 2024-03-25T18:00:07.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.502+0000 7f7c5524a640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-25T18:00:07.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.502+0000 7f7c5524a640 10 client.4381.objecter successfully canceled tick 2024-03-25T18:00:07.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.502+0000 7f7c5524a640 1 -- 172.21.15.18:0/2992698960 >> v1:172.21.15.18:6800/3721244451 conn(0x7f7c3c06dbf0 legacy=0x7f7c3c0700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:07.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.502+0000 7f7c5524a640 10 monclient: shutdown 2024-03-25T18:00:07.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.502+0000 7f7c5524a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:00:07.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.502+0000 7f7c5524a640 1 -- 172.21.15.18:0/2992698960 >> v1:172.21.15.169:6789/0 conn(0x7f7c5013f710 legacy=0x7f7c50130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:07.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.503+0000 7f7c5524a640 1 -- 172.21.15.18:0/2992698960 shutdown_connections 2024-03-25T18:00:07.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.503+0000 7f7c5524a640 1 -- 172.21.15.18:0/2992698960 >> 172.21.15.18:0/2992698960 conn(0x7f7c500b7ea0 msgr2=0x7f7c50074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:00:07.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.503+0000 7f7c5524a640 1 -- 172.21.15.18:0/2992698960 shutdown_connections 2024-03-25T18:00:07.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.503+0000 7f7c5524a640 1 -- 172.21.15.18:0/2992698960 wait complete. 2024-03-25T18:00:07.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:07.503+0000 7f7c5524a640 1 librados: shutdown 2024-03-25T18:00:07.517 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:00:14.323 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:00:14.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.442+0000 7fe664708640 10 monclient: get_monmap_and_config 2024-03-25T18:00:14.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.442+0000 7fe664708640 10 monclient: build_initial_monmap 2024-03-25T18:00:14.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.442+0000 7fe664708640 10 monclient: monmap: 2024-03-25T18:00:14.443 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:00:14.443 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:14.443 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:00:14.443833+0000 2024-03-25T18:00:14.444 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:00:14.443833+0000 2024-03-25T18:00:14.444 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:00:14.444 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:14.444 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:00:14.444 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:00:14.444 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:00:14.444 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.443+0000 7fe664708640 1 Processor -- start 2024-03-25T18:00:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.443+0000 7fe664708640 1 -- start start 2024-03-25T18:00:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.443+0000 7fe664708640 10 monclient: init 2024-03-25T18:00:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.443+0000 7fe664708640 10 monclient: _reopen_session rank -1 2024-03-25T18:00:14.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.444+0000 7fe664708640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:00:14.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.444+0000 7fe664708640 10 monclient(hunting): picked mon.noname-b con 0x7fe65c0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:00:14.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.444+0000 7fe664708640 10 monclient(hunting): picked mon.noname-c con 0x7fe65c13f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:00:14.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.444+0000 7fe664708640 10 monclient(hunting): picked mon.noname-a con 0x7fe65c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:00:14.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.444+0000 7fe664708640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65c142520 con 0x7fe65c1430c0 2024-03-25T18:00:14.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.444+0000 7fe664708640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65c0b28a0 con 0x7fe65c13f710 2024-03-25T18:00:14.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.444+0000 7fe664708640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65c0b2a30 con 0x7fe65c0a98f0 2024-03-25T18:00:14.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.444+0000 7fe664708640 10 monclient(hunting): _renew_subs 2024-03-25T18:00:14.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.444+0000 7fe664708640 10 monclient(hunting): authenticate will time out at 7143.022757s 2024-03-25T18:00:14.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.451+0000 7fe662ca5640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fe65c1430c0 0x7fe65c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:44364/0 (socket says 172.21.15.18:44364) 2024-03-25T18:00:14.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.451+0000 7fe662ca5640 1 -- 172.21.15.18:0/135555480 learned_addr learned my addr 172.21.15.18:0/135555480 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:00:14.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.462+0000 7fe6614a2640 1 -- 172.21.15.18:0/135555480 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861472586 0 0) 0x7fe65c0b28a0 con 0x7fe65c13f710 2024-03-25T18:00:14.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.462+0000 7fe6614a2640 10 monclient(hunting): _negotiate 2024-03-25T18:00:14.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.462+0000 7fe6614a2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:14.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.462+0000 7fe6614a2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:14.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.462+0000 7fe6614a2640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:00:14.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.462+0000 7fe6614a2640 1 -- 172.21.15.18:0/135555480 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe648003520 con 0x7fe65c13f710 2024-03-25T18:00:14.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.469+0000 7fe6614a2640 1 -- 172.21.15.18:0/135555480 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977566898 0 0) 0x7fe65c142520 con 0x7fe65c1430c0 2024-03-25T18:00:14.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.469+0000 7fe6614a2640 10 monclient(hunting): _negotiate 2024-03-25T18:00:14.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.469+0000 7fe6614a2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:14.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.469+0000 7fe6614a2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:14.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.469+0000 7fe6614a2640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:00:14.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.469+0000 7fe6614a2640 1 -- 172.21.15.18:0/135555480 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe65c0b28a0 con 0x7fe65c1430c0 2024-03-25T18:00:14.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.475+0000 7fe6614a2640 1 -- 172.21.15.18:0/135555480 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2097307777 0 0) 0x7fe648003520 con 0x7fe65c13f710 2024-03-25T18:00:14.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.475+0000 7fe6614a2640 1 -- 172.21.15.18:0/135555480 >> v1:172.21.15.18:6790/0 conn(0x7fe65c0a98f0 legacy=0x7fe65c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:14.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.475+0000 7fe6614a2640 1 -- 172.21.15.18:0/135555480 >> v1:172.21.15.18:6789/0 conn(0x7fe65c1430c0 legacy=0x7fe65c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:14.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.475+0000 7fe6614a2640 10 monclient: _finish_hunting 0 2024-03-25T18:00:14.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.475+0000 7fe6614a2640 1 monclient: found mon.noname-c 2024-03-25T18:00:14.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.475+0000 7fe6614a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:00:14.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.475+0000 7fe6614a2640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:00:14.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.475+0000 7fe6614a2640 1 -- 172.21.15.18:0/135555480 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c0b44c0 con 0x7fe65c13f710 2024-03-25T18:00:14.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.475+0000 7fe6614a2640 10 monclient: _finish_auth 0 2024-03-25T18:00:14.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.475+0000 7fe6614a2640 10 monclient: _check_auth_tickets 2024-03-25T18:00:14.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.475+0000 7fe6614a2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:00:14.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.475+0000 7fe6614a2640 1 -- 172.21.15.18:0/135555480 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fe64c001690 con 0x7fe65c13f710 2024-03-25T18:00:14.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.475+0000 7fe6614a2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:14.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.475+0000 7fe6614a2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:00:14.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.475+0000 7fe6614a2640 10 monclient: dump: 2024-03-25T18:00:14.477 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:14.477 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:14.477 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:14.477 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:14.477 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:14.477 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:14.478 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:14.478 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:14.478 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:14.478 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:14.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.475+0000 7fe664708640 5 monclient: authenticate success, global_id 4384 2024-03-25T18:00:14.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.475+0000 7fe664708640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:00:14.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.481+0000 7fe6614a2640 1 -- 172.21.15.18:0/135555480 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe64c0017f0 con 0x7fe65c13f710 2024-03-25T18:00:14.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.481+0000 7fe6614a2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:00:14.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.481+0000 7fe6614a2640 1 -- 172.21.15.18:0/135555480 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fe64c001c00 con 0x7fe65c13f710 2024-03-25T18:00:14.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.481+0000 7fe664708640 10 monclient: get_monmap_and_config success 2024-03-25T18:00:14.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.481+0000 7fe664708640 4 set_mon_vals no callback set 2024-03-25T18:00:14.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.481+0000 7fe6614a2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:14.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.481+0000 7fe6614a2640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:00:14.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.481+0000 7fe6614a2640 10 monclient: dump: 2024-03-25T18:00:14.482 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:14.482 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:14.482 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:14.482 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:14.482 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:14.482 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:14.482 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:14.482 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:14.483 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:14.483 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:14.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.481+0000 7fe664708640 10 monclient: shutdown 2024-03-25T18:00:14.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.481+0000 7fe664708640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:00:14.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.481+0000 7fe664708640 1 -- 172.21.15.18:0/135555480 >> v1:172.21.15.169:6789/0 conn(0x7fe65c13f710 legacy=0x7fe65c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:14.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.482+0000 7fe664708640 1 -- 172.21.15.18:0/135555480 shutdown_connections 2024-03-25T18:00:14.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.482+0000 7fe664708640 1 -- 172.21.15.18:0/135555480 >> 172.21.15.18:0/135555480 conn(0x7fe65c0b7ea0 msgr2=0x7fe65c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:00:14.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.482+0000 7fe664708640 1 -- 172.21.15.18:0/135555480 shutdown_connections 2024-03-25T18:00:14.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.482+0000 7fe664708640 1 -- 172.21.15.18:0/135555480 wait complete. 2024-03-25T18:00:14.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.482+0000 7fe664708640 10 monclient: build_initial_monmap 2024-03-25T18:00:14.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.482+0000 7fe664708640 10 monclient: monmap: 2024-03-25T18:00:14.485 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:00:14.485 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:00:14.485 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:00:14.486 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:00:14.486 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:00:14.486 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:00:14.486 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:00:14.486 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:00:14.486 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:00:14.486 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:14.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.483+0000 7fe664708640 1 librados: starting msgr at 2024-03-25T18:00:14.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.483+0000 7fe664708640 1 librados: starting objecter 2024-03-25T18:00:14.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.483+0000 7fe664708640 1 Processor -- start 2024-03-25T18:00:14.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.483+0000 7fe664708640 1 -- start start 2024-03-25T18:00:14.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.483+0000 7fe664708640 1 librados: setting wanted keys 2024-03-25T18:00:14.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.483+0000 7fe664708640 1 librados: calling monclient init 2024-03-25T18:00:14.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.483+0000 7fe664708640 10 monclient: init 2024-03-25T18:00:14.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.483+0000 7fe664708640 10 monclient: _reopen_session rank -1 2024-03-25T18:00:14.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.484+0000 7fe664708640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:00:14.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.484+0000 7fe664708640 10 monclient(hunting): picked mon.noname-b con 0x7fe65c0a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:00:14.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.484+0000 7fe664708640 10 monclient(hunting): picked mon.noname-c con 0x7fe65c13f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:00:14.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.484+0000 7fe664708640 10 monclient(hunting): picked mon.noname-a con 0x7fe65c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:00:14.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.484+0000 7fe664708640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65c138a50 con 0x7fe65c1430c0 2024-03-25T18:00:14.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.484+0000 7fe664708640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65c0ab9e0 con 0x7fe65c0a98f0 2024-03-25T18:00:14.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.484+0000 7fe664708640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65c1d7880 con 0x7fe65c13f710 2024-03-25T18:00:14.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.484+0000 7fe664708640 10 monclient(hunting): _renew_subs 2024-03-25T18:00:14.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.484+0000 7fe664708640 10 monclient(hunting): authenticate will time out at 7143.062869s 2024-03-25T18:00:14.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.491+0000 7fe662ca5640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fe65c1430c0 0x7fe65c1e1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:44392/0 (socket says 172.21.15.18:44392) 2024-03-25T18:00:14.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.491+0000 7fe662ca5640 1 -- 172.21.15.18:0/2926323944 learned_addr learned my addr 172.21.15.18:0/2926323944 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:00:14.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.495+0000 7fe652ffd640 10 client.?.objecter ms_handle_connect 0x7fe65c1430c0 2024-03-25T18:00:14.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.495+0000 7fe652ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:14.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.500+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308934088 0 0) 0x7fe65c138a50 con 0x7fe65c1430c0 2024-03-25T18:00:14.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.500+0000 7fe652ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:00:14.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.500+0000 7fe652ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:14.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.500+0000 7fe652ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:14.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.500+0000 7fe652ffd640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:00:14.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.500+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe640004eb0 con 0x7fe65c1430c0 2024-03-25T18:00:14.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.507+0000 7fe652ffd640 10 client.?.objecter ms_handle_connect 0x7fe65c13f710 2024-03-25T18:00:14.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.507+0000 7fe652ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:14.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.508+0000 7fe652ffd640 10 client.?.objecter ms_handle_connect 0x7fe65c0a98f0 2024-03-25T18:00:14.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.508+0000 7fe652ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:14.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.510+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 989800354 0 0) 0x7fe640004eb0 con 0x7fe65c1430c0 2024-03-25T18:00:14.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.510+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe65c138a50 con 0x7fe65c1430c0 2024-03-25T18:00:14.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.510+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fe654001c10 con 0x7fe65c1430c0 2024-03-25T18:00:14.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.510+0000 7fe652ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:14.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.510+0000 7fe652ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:00:14.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.510+0000 7fe652ffd640 10 monclient(hunting): dump: 2024-03-25T18:00:14.511 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:14.511 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:14.511 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:14.512 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:14.512 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:14.512 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:14.512 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:14.512 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:14.512 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:14.512 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:14.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.510+0000 7fe652ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:00:14.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.517+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400394578 0 0) 0x7fe65c1d7880 con 0x7fe65c13f710 2024-03-25T18:00:14.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.517+0000 7fe652ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:00:14.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.517+0000 7fe652ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:14.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.517+0000 7fe652ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:14.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.517+0000 7fe652ffd640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:00:14.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.517+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe640004eb0 con 0x7fe65c13f710 2024-03-25T18:00:14.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.518+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596771112 0 0) 0x7fe65c0ab9e0 con 0x7fe65c0a98f0 2024-03-25T18:00:14.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.518+0000 7fe652ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:00:14.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.518+0000 7fe652ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:14.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.518+0000 7fe652ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:14.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.518+0000 7fe652ffd640 10 monclient(hunting): my global_id is 4387 2024-03-25T18:00:14.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.518+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe65c1d7880 con 0x7fe65c0a98f0 2024-03-25T18:00:14.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.524+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2775860156 0 0) 0x7fe65c138a50 con 0x7fe65c1430c0 2024-03-25T18:00:14.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.524+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 >> v1:172.21.15.18:6790/0 conn(0x7fe65c13f710 legacy=0x7fe65c1d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:14.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.524+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 >> v1:172.21.15.169:6789/0 conn(0x7fe65c0a98f0 legacy=0x7fe65c138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:14.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.524+0000 7fe652ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:00:14.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.524+0000 7fe652ffd640 1 monclient: found mon.a 2024-03-25T18:00:14.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.524+0000 7fe652ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:00:14.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.524+0000 7fe652ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:00:14.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.524+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c1d84b0 con 0x7fe65c1430c0 2024-03-25T18:00:14.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.524+0000 7fe652ffd640 10 monclient: _finish_auth 0 2024-03-25T18:00:14.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.524+0000 7fe652ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:00:14.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.524+0000 7fe652ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:00:14.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.524+0000 7fe664708640 5 monclient: authenticate success, global_id 4398 2024-03-25T18:00:14.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.524+0000 7fe664708640 10 monclient: _renew_subs 2024-03-25T18:00:14.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.524+0000 7fe664708640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:00:14.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.524+0000 7fe664708640 1 -- 172.21.15.18:0/2926323944 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65c1d51f0 con 0x7fe65c1430c0 2024-03-25T18:00:14.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.525+0000 7fe664708640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:00:14.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.525+0000 7fe664708640 10 monclient: _renew_subs 2024-03-25T18:00:14.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.525+0000 7fe664708640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:00:14.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.525+0000 7fe664708640 1 -- 172.21.15.18:0/2926323944 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe65c138e60 con 0x7fe65c1430c0 2024-03-25T18:00:14.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.525+0000 7fe664708640 1 librados: init done 2024-03-25T18:00:14.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.525+0000 7fe664708640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:14.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.525+0000 7fe664708640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:14.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.525+0000 7fe664708640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:00:14.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.525+0000 7fe664708640 1 -- 172.21.15.18:0/2926323944 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65c1d76e0 con 0x7fe65c1430c0 2024-03-25T18:00:14.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.846+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe654004b60 con 0x7fe65c1430c0 2024-03-25T18:00:14.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.846+0000 7fe652ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:00:14.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.846+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fe654004cd0 con 0x7fe65c1430c0 2024-03-25T18:00:14.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.846+0000 7fe652ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:14.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.846+0000 7fe652ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:00:14.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.846+0000 7fe652ffd640 10 monclient: dump: 2024-03-25T18:00:14.848 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:14.848 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:14.848 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:14.848 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:14.848 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:14.849 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:14.849 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:14.849 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:14.849 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:14.849 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:14.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.846+0000 7fe660ca1640 4 set_mon_vals no callback set 2024-03-25T18:00:14.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.847+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fe65401aca0 con 0x7fe65c1430c0 2024-03-25T18:00:14.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.848+0000 7fe6624a4640 10 monclient: get_auth_request con 0x7fe64006de80 auth_method 0 2024-03-25T18:00:14.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.848+0000 7fe652ffd640 10 client.4398.objecter ms_handle_connect 0x7fe64006de80 2024-03-25T18:00:14.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.855+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fe654054990 con 0x7fe65c1430c0 2024-03-25T18:00:14.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.855+0000 7fe652ffd640 10 client.4398.objecter ms_dispatch 0x7fe65c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:00:14.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.855+0000 7fe652ffd640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:00:14.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.855+0000 7fe652ffd640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:00:14.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.855+0000 7fe652ffd640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-25T18:00:14.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.855+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe6540504a0 con 0x7fe65c1430c0 2024-03-25T18:00:14.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.855+0000 7fe652ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:14.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:14.855+0000 7fe652ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:00:15.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.093+0000 7fe664708640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:15.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.093+0000 7fe664708640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:15.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.093+0000 7fe664708640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:00:15.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.093+0000 7fe664708640 1 -- 172.21.15.18:0/2926323944 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe65c1d7a10 con 0x7fe65c1430c0 2024-03-25T18:00:15.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.129+0000 7fe652ffd640 1 -- 172.21.15.18:0/2926323944 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fe6540523d0 con 0x7fe65c1430c0 2024-03-25T18:00:15.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.129+0000 7fe652ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:15.130 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.129+0000 7fe652ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:00:15.130 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:00:15.130 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:00:15.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.131+0000 7fe664708640 10 librados: watch_flush enter 2024-03-25T18:00:15.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.131+0000 7fe664708640 10 librados: watch_flush exit 2024-03-25T18:00:15.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.131+0000 7fe664708640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-25T18:00:15.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.131+0000 7fe664708640 10 client.4398.objecter successfully canceled tick 2024-03-25T18:00:15.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.131+0000 7fe664708640 1 -- 172.21.15.18:0/2926323944 >> v1:172.21.15.18:6800/3721244451 conn(0x7fe64006de80 legacy=0x7fe640070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:15.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.131+0000 7fe664708640 10 monclient: shutdown 2024-03-25T18:00:15.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.131+0000 7fe664708640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:00:15.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.131+0000 7fe664708640 1 -- 172.21.15.18:0/2926323944 >> v1:172.21.15.18:6789/0 conn(0x7fe65c1430c0 legacy=0x7fe65c1e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:15.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.131+0000 7fe664708640 1 -- 172.21.15.18:0/2926323944 shutdown_connections 2024-03-25T18:00:15.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.131+0000 7fe664708640 1 -- 172.21.15.18:0/2926323944 >> 172.21.15.18:0/2926323944 conn(0x7fe65c0b7ea0 msgr2=0x7fe65c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:00:15.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.131+0000 7fe664708640 1 -- 172.21.15.18:0/2926323944 shutdown_connections 2024-03-25T18:00:15.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.131+0000 7fe664708640 1 -- 172.21.15.18:0/2926323944 wait complete. 2024-03-25T18:00:15.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:15.131+0000 7fe664708640 1 librados: shutdown 2024-03-25T18:00:15.142 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:00:21.948 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:00:22.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.063+0000 7f8ba8782640 10 monclient: get_monmap_and_config 2024-03-25T18:00:22.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.063+0000 7f8ba8782640 10 monclient: build_initial_monmap 2024-03-25T18:00:22.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.063+0000 7f8ba8782640 10 monclient: monmap: 2024-03-25T18:00:22.065 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:00:22.065 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:22.065 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:00:22.065005+0000 2024-03-25T18:00:22.065 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:00:22.065005+0000 2024-03-25T18:00:22.065 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:00:22.065 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:22.066 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:00:22.066 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:00:22.066 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:00:22.066 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:22.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.064+0000 7f8ba8782640 1 Processor -- start 2024-03-25T18:00:22.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.064+0000 7f8ba8782640 1 -- start start 2024-03-25T18:00:22.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.064+0000 7f8ba8782640 10 monclient: init 2024-03-25T18:00:22.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.065+0000 7f8ba8782640 10 monclient: _reopen_session rank -1 2024-03-25T18:00:22.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.065+0000 7f8ba8782640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:00:22.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.065+0000 7f8ba8782640 10 monclient(hunting): picked mon.noname-a con 0x7f8ba00a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:00:22.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.065+0000 7f8ba8782640 10 monclient(hunting): picked mon.noname-c con 0x7f8ba013f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:00:22.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.065+0000 7f8ba8782640 10 monclient(hunting): picked mon.noname-b con 0x7f8ba01430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:00:22.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.065+0000 7f8ba8782640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba0142520 con 0x7f8ba00a98f0 2024-03-25T18:00:22.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.065+0000 7f8ba8782640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba00b28a0 con 0x7f8ba013f710 2024-03-25T18:00:22.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.065+0000 7f8ba8782640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba00b2a30 con 0x7f8ba01430c0 2024-03-25T18:00:22.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.065+0000 7f8ba8782640 10 monclient(hunting): _renew_subs 2024-03-25T18:00:22.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.065+0000 7f8ba8782640 10 monclient(hunting): authenticate will time out at 7150.644458s 2024-03-25T18:00:22.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.071+0000 7f8ba651e640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f8ba00a98f0 0x7f8ba00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:48824/0 (socket says 172.21.15.18:48824) 2024-03-25T18:00:22.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.071+0000 7f8ba651e640 1 -- 172.21.15.18:0/3951228821 learned_addr learned my addr 172.21.15.18:0/3951228821 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:00:22.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.088+0000 7f8ba551c640 1 -- 172.21.15.18:0/3951228821 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288427229 0 0) 0x7f8ba00b2a30 con 0x7f8ba01430c0 2024-03-25T18:00:22.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.088+0000 7f8ba551c640 10 monclient(hunting): _negotiate 2024-03-25T18:00:22.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.088+0000 7f8ba551c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:22.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.088+0000 7f8ba551c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:22.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.088+0000 7f8ba551c640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:00:22.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.088+0000 7f8ba551c640 1 -- 172.21.15.18:0/3951228821 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b8c003520 con 0x7f8ba01430c0 2024-03-25T18:00:22.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.091+0000 7f8ba551c640 1 -- 172.21.15.18:0/3951228821 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924439178 0 0) 0x7f8ba0142520 con 0x7f8ba00a98f0 2024-03-25T18:00:22.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.091+0000 7f8ba551c640 10 monclient(hunting): _negotiate 2024-03-25T18:00:22.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.091+0000 7f8ba551c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:22.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.091+0000 7f8ba551c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:22.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.091+0000 7f8ba551c640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:00:22.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.091+0000 7f8ba551c640 1 -- 172.21.15.18:0/3951228821 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ba00b2a30 con 0x7f8ba00a98f0 2024-03-25T18:00:22.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.097+0000 7f8ba551c640 1 -- 172.21.15.18:0/3951228821 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350187518 0 0) 0x7f8ba00b28a0 con 0x7f8ba013f710 2024-03-25T18:00:22.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.097+0000 7f8ba551c640 10 monclient(hunting): _negotiate 2024-03-25T18:00:22.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.097+0000 7f8ba551c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:22.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.097+0000 7f8ba551c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:22.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.097+0000 7f8ba551c640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:00:22.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.097+0000 7f8ba551c640 1 -- 172.21.15.18:0/3951228821 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ba0142520 con 0x7f8ba013f710 2024-03-25T18:00:22.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.104+0000 7f8ba551c640 1 -- 172.21.15.18:0/3951228821 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1240606229 0 0) 0x7f8ba0142520 con 0x7f8ba013f710 2024-03-25T18:00:22.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.104+0000 7f8ba551c640 1 -- 172.21.15.18:0/3951228821 >> v1:172.21.15.18:6790/0 conn(0x7f8ba01430c0 legacy=0x7f8ba014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:22.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.104+0000 7f8ba551c640 1 -- 172.21.15.18:0/3951228821 >> v1:172.21.15.18:6789/0 conn(0x7f8ba00a98f0 legacy=0x7f8ba00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:22.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.104+0000 7f8ba551c640 10 monclient: _finish_hunting 0 2024-03-25T18:00:22.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.104+0000 7f8ba551c640 1 monclient: found mon.noname-c 2024-03-25T18:00:22.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.104+0000 7f8ba551c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:00:22.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.104+0000 7f8ba551c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:00:22.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.104+0000 7f8ba551c640 1 -- 172.21.15.18:0/3951228821 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba00b44c0 con 0x7f8ba013f710 2024-03-25T18:00:22.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.104+0000 7f8ba551c640 10 monclient: _finish_auth 0 2024-03-25T18:00:22.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.104+0000 7f8ba551c640 10 monclient: _check_auth_tickets 2024-03-25T18:00:22.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.104+0000 7f8ba551c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:00:22.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.104+0000 7f8ba8782640 5 monclient: authenticate success, global_id 4390 2024-03-25T18:00:22.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.104+0000 7f8ba8782640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:00:22.108 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.107+0000 7f8ba551c640 1 -- 172.21.15.18:0/3951228821 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8b900014f0 con 0x7f8ba013f710 2024-03-25T18:00:22.108 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.107+0000 7f8ba551c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:22.108 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.107+0000 7f8ba551c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:00:22.108 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.107+0000 7f8ba551c640 10 monclient: dump: 2024-03-25T18:00:22.108 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:22.108 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:22.109 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:22.109 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:22.109 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:22.109 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:22.109 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:22.109 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:22.109 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:22.110 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:22.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.107+0000 7f8ba8782640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:00:22.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.116+0000 7f8ba551c640 1 -- 172.21.15.18:0/3951228821 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b900017c0 con 0x7f8ba013f710 2024-03-25T18:00:22.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.116+0000 7f8ba551c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:00:22.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.116+0000 7f8ba551c640 1 -- 172.21.15.18:0/3951228821 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8b90001b50 con 0x7f8ba013f710 2024-03-25T18:00:22.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.116+0000 7f8ba8782640 10 monclient: get_monmap_and_config success 2024-03-25T18:00:22.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.116+0000 7f8ba8782640 4 set_mon_vals no callback set 2024-03-25T18:00:22.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.116+0000 7f8ba551c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:22.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.116+0000 7f8ba551c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:00:22.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.116+0000 7f8ba551c640 10 monclient: dump: 2024-03-25T18:00:22.118 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:22.119 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:22.119 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:22.119 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:22.119 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:22.119 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:22.119 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:22.119 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:22.120 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:22.120 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:22.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.117+0000 7f8ba8782640 10 monclient: shutdown 2024-03-25T18:00:22.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.117+0000 7f8ba8782640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:00:22.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.117+0000 7f8ba8782640 1 -- 172.21.15.18:0/3951228821 >> v1:172.21.15.169:6789/0 conn(0x7f8ba013f710 legacy=0x7f8ba01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:22.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.117+0000 7f8ba8782640 1 -- 172.21.15.18:0/3951228821 shutdown_connections 2024-03-25T18:00:22.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.117+0000 7f8ba8782640 1 -- 172.21.15.18:0/3951228821 >> 172.21.15.18:0/3951228821 conn(0x7f8ba00b7ea0 msgr2=0x7f8ba00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:00:22.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.117+0000 7f8ba8782640 1 -- 172.21.15.18:0/3951228821 shutdown_connections 2024-03-25T18:00:22.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.117+0000 7f8ba8782640 1 -- 172.21.15.18:0/3951228821 wait complete. 2024-03-25T18:00:22.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.118+0000 7f8ba8782640 10 monclient: build_initial_monmap 2024-03-25T18:00:22.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.118+0000 7f8ba8782640 10 monclient: monmap: 2024-03-25T18:00:22.124 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:00:22.125 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:00:22.125 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:00:22.125 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:00:22.125 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:00:22.125 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:00:22.125 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:00:22.125 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:00:22.126 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:00:22.126 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:22.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.118+0000 7f8ba8782640 1 librados: starting msgr at 2024-03-25T18:00:22.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.118+0000 7f8ba8782640 1 librados: starting objecter 2024-03-25T18:00:22.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.118+0000 7f8ba8782640 1 Processor -- start 2024-03-25T18:00:22.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.118+0000 7f8ba8782640 1 -- start start 2024-03-25T18:00:22.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.118+0000 7f8ba8782640 1 librados: setting wanted keys 2024-03-25T18:00:22.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.118+0000 7f8ba8782640 1 librados: calling monclient init 2024-03-25T18:00:22.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.118+0000 7f8ba8782640 10 monclient: init 2024-03-25T18:00:22.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.118+0000 7f8ba8782640 10 monclient: _reopen_session rank -1 2024-03-25T18:00:22.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.118+0000 7f8ba8782640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:00:22.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.119+0000 7f8ba8782640 10 monclient(hunting): picked mon.noname-a con 0x7f8ba00a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:00:22.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.119+0000 7f8ba8782640 10 monclient(hunting): picked mon.noname-b con 0x7f8ba013f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:00:22.127 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.119+0000 7f8ba8782640 10 monclient(hunting): picked mon.noname-c con 0x7f8ba01430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:00:22.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.119+0000 7f8ba8782640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba00abae0 con 0x7f8ba00a98f0 2024-03-25T18:00:22.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.119+0000 7f8ba8782640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba00acb40 con 0x7f8ba013f710 2024-03-25T18:00:22.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.119+0000 7f8ba8782640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba00abd30 con 0x7f8ba01430c0 2024-03-25T18:00:22.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.119+0000 7f8ba8782640 10 monclient(hunting): _renew_subs 2024-03-25T18:00:22.128 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.119+0000 7f8ba8782640 10 monclient(hunting): authenticate will time out at 7150.697728s 2024-03-25T18:00:22.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.132+0000 7f8ba5d1d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f8ba013f710 0x7f8ba00ac430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:45938/0 (socket says 172.21.15.18:45938) 2024-03-25T18:00:22.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.132+0000 7f8ba5d1d640 1 -- 172.21.15.18:0/576215209 learned_addr learned my addr 172.21.15.18:0/576215209 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:00:22.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.136+0000 7f8b96ffd640 10 client.?.objecter ms_handle_connect 0x7f8ba013f710 2024-03-25T18:00:22.137 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.136+0000 7f8b96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:22.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.140+0000 7f8b96ffd640 10 client.?.objecter ms_handle_connect 0x7f8ba01430c0 2024-03-25T18:00:22.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.140+0000 7f8b96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:22.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.150+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814773301 0 0) 0x7f8ba00acb40 con 0x7f8ba013f710 2024-03-25T18:00:22.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.150+0000 7f8b96ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:00:22.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.150+0000 7f8b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:22.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.150+0000 7f8b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:22.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.150+0000 7f8b96ffd640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:00:22.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.150+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b84004eb0 con 0x7f8ba013f710 2024-03-25T18:00:22.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.155+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572319247 0 0) 0x7f8ba00abd30 con 0x7f8ba01430c0 2024-03-25T18:00:22.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.155+0000 7f8b96ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:00:22.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.155+0000 7f8b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:22.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.155+0000 7f8b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:22.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.155+0000 7f8b96ffd640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:00:22.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.155+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ba00acb40 con 0x7f8ba01430c0 2024-03-25T18:00:22.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.167+0000 7f8b96ffd640 10 client.?.objecter ms_handle_connect 0x7f8ba00a98f0 2024-03-25T18:00:22.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.167+0000 7f8b96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:22.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.176+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1622623910 0 0) 0x7f8b84004eb0 con 0x7f8ba013f710 2024-03-25T18:00:22.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.176+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ba00abd30 con 0x7f8ba013f710 2024-03-25T18:00:22.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.176+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8b90001630 con 0x7f8ba013f710 2024-03-25T18:00:22.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.176+0000 7f8b96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:22.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.176+0000 7f8b96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:00:22.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.176+0000 7f8b96ffd640 10 monclient(hunting): dump: 2024-03-25T18:00:22.178 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:22.178 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:22.178 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:22.178 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:22.178 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:22.178 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:22.178 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:22.178 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:22.178 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:22.178 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:22.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.176+0000 7f8b96ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:00:22.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.182+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221897994 0 0) 0x7f8ba00abae0 con 0x7f8ba00a98f0 2024-03-25T18:00:22.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.182+0000 7f8b96ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:00:22.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.182+0000 7f8b96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:22.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.182+0000 7f8b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:22.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.182+0000 7f8b96ffd640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:00:22.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.182+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b84004eb0 con 0x7f8ba00a98f0 2024-03-25T18:00:22.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.195+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3684974008 0 0) 0x7f8b84004eb0 con 0x7f8ba00a98f0 2024-03-25T18:00:22.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.195+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ba00abae0 con 0x7f8ba00a98f0 2024-03-25T18:00:22.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.195+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8b9c001930 con 0x7f8ba00a98f0 2024-03-25T18:00:22.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.195+0000 7f8b96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:22.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.195+0000 7f8b96ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:00:22.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.195+0000 7f8b96ffd640 10 monclient(hunting): dump: 2024-03-25T18:00:22.197 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:22.197 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:22.197 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:22.197 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:22.197 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:22.197 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:22.197 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:22.197 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:22.197 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:22.197 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:22.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.196+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2066022763 0 0) 0x7f8ba00acb40 con 0x7f8ba01430c0 2024-03-25T18:00:22.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.196+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b84004eb0 con 0x7f8ba01430c0 2024-03-25T18:00:22.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.196+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8b98001b30 con 0x7f8ba01430c0 2024-03-25T18:00:22.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.196+0000 7f8b96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:22.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.196+0000 7f8b96ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:00:22.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.196+0000 7f8b96ffd640 10 monclient(hunting): dump: 2024-03-25T18:00:22.198 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:22.199 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:22.199 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:22.199 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:22.199 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:22.199 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:22.199 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:22.199 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:22.199 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:22.199 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:22.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.199+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1902383402 0 0) 0x7f8ba00abd30 con 0x7f8ba013f710 2024-03-25T18:00:22.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.199+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 >> v1:172.21.15.18:6790/0 conn(0x7f8ba01430c0 legacy=0x7f8ba01e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:22.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.199+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 >> v1:172.21.15.18:6789/0 conn(0x7f8ba00a98f0 legacy=0x7f8ba01388d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:22.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.199+0000 7f8b96ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:00:22.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.199+0000 7f8b96ffd640 1 monclient: found mon.b 2024-03-25T18:00:22.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.199+0000 7f8b96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:00:22.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.199+0000 7f8b96ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:22.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.199+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba00ac010 con 0x7f8ba013f710 2024-03-25T18:00:22.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.199+0000 7f8b96ffd640 10 monclient: _finish_auth 0 2024-03-25T18:00:22.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.199+0000 7f8b96ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:00:22.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.199+0000 7f8b96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:00:22.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.200+0000 7f8ba8782640 5 monclient: authenticate success, global_id 4393 2024-03-25T18:00:22.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.200+0000 7f8ba8782640 10 monclient: _renew_subs 2024-03-25T18:00:22.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.200+0000 7f8ba8782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:22.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.200+0000 7f8ba8782640 1 -- 172.21.15.18:0/576215209 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba0139110 con 0x7f8ba013f710 2024-03-25T18:00:22.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.200+0000 7f8ba8782640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:00:22.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.200+0000 7f8ba8782640 10 monclient: _renew_subs 2024-03-25T18:00:22.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.200+0000 7f8ba8782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:22.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.200+0000 7f8ba8782640 1 -- 172.21.15.18:0/576215209 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba0136560 con 0x7f8ba013f710 2024-03-25T18:00:22.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.200+0000 7f8ba8782640 1 librados: init done 2024-03-25T18:00:22.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.201+0000 7f8ba8782640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:22.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.201+0000 7f8ba8782640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:22.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.201+0000 7f8ba8782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:22.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.201+0000 7f8ba8782640 1 -- 172.21.15.18:0/576215209 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba00627b0 con 0x7f8ba013f710 2024-03-25T18:00:22.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.514+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b90001200 con 0x7f8ba013f710 2024-03-25T18:00:22.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.514+0000 7f8b96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:00:22.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.514+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8b90003b00 con 0x7f8ba013f710 2024-03-25T18:00:22.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.514+0000 7f8b96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:22.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.514+0000 7f8b96ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:00:22.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.514+0000 7f8b96ffd640 10 monclient: dump: 2024-03-25T18:00:22.516 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:22.516 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:22.516 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:22.516 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:22.516 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:22.516 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:22.516 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:22.516 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:22.516 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:22.517 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:22.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.514+0000 7f8ba4d1b640 4 set_mon_vals no callback set 2024-03-25T18:00:22.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.515+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f8b90003d20 con 0x7f8ba013f710 2024-03-25T18:00:22.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.516+0000 7f8ba651e640 10 monclient: get_auth_request con 0x7f8b8406e110 auth_method 0 2024-03-25T18:00:22.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.516+0000 7f8b96ffd640 10 client.4393.objecter ms_handle_connect 0x7f8b8406e110 2024-03-25T18:00:22.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.521+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f8b90054910 con 0x7f8ba013f710 2024-03-25T18:00:22.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.521+0000 7f8b96ffd640 10 client.4393.objecter ms_dispatch 0x7f8ba0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:00:22.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.521+0000 7f8b96ffd640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:00:22.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.521+0000 7f8b96ffd640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:00:22.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.521+0000 7f8b96ffd640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-25T18:00:22.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.521+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b90050250 con 0x7f8ba013f710 2024-03-25T18:00:22.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.521+0000 7f8b96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:22.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.521+0000 7f8b96ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:00:22.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.750+0000 7f8ba8782640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:22.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.750+0000 7f8ba8782640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:22.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.750+0000 7f8ba8782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:22.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.750+0000 7f8ba8782640 1 -- 172.21.15.18:0/576215209 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ba01392a0 con 0x7f8ba013f710 2024-03-25T18:00:22.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.834+0000 7f8b96ffd640 1 -- 172.21.15.18:0/576215209 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f8b90052350 con 0x7f8ba013f710 2024-03-25T18:00:22.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.834+0000 7f8b96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:22.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.834+0000 7f8b96ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:00:22.836 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:00:22.836 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:00:22.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.837+0000 7f8ba8782640 10 librados: watch_flush enter 2024-03-25T18:00:22.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.837+0000 7f8ba8782640 10 librados: watch_flush exit 2024-03-25T18:00:22.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.837+0000 7f8ba8782640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-25T18:00:22.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.837+0000 7f8ba8782640 10 client.4393.objecter successfully canceled tick 2024-03-25T18:00:22.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.837+0000 7f8ba8782640 1 -- 172.21.15.18:0/576215209 >> v1:172.21.15.18:6800/3721244451 conn(0x7f8b8406e110 legacy=0x7f8b840705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:22.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.837+0000 7f8ba8782640 10 monclient: shutdown 2024-03-25T18:00:22.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.837+0000 7f8ba8782640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:00:22.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.837+0000 7f8ba8782640 1 -- 172.21.15.18:0/576215209 >> v1:172.21.15.169:6789/0 conn(0x7f8ba013f710 legacy=0x7f8ba00ac430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:22.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.838+0000 7f8ba8782640 1 -- 172.21.15.18:0/576215209 shutdown_connections 2024-03-25T18:00:22.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.838+0000 7f8ba8782640 1 -- 172.21.15.18:0/576215209 >> 172.21.15.18:0/576215209 conn(0x7f8ba00b7ea0 msgr2=0x7f8ba0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:00:22.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.838+0000 7f8ba8782640 1 -- 172.21.15.18:0/576215209 shutdown_connections 2024-03-25T18:00:22.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.838+0000 7f8ba8782640 1 -- 172.21.15.18:0/576215209 wait complete. 2024-03-25T18:00:22.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:22.838+0000 7f8ba8782640 1 librados: shutdown 2024-03-25T18:00:22.850 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:00:29.657 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:00:29.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.772+0000 7f14c81b5640 10 monclient: get_monmap_and_config 2024-03-25T18:00:29.773 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.772+0000 7f14c81b5640 10 monclient: build_initial_monmap 2024-03-25T18:00:29.773 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.772+0000 7f14c81b5640 10 monclient: monmap: 2024-03-25T18:00:29.773 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:00:29.773 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:29.773 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:00:29.773418+0000 2024-03-25T18:00:29.773 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:00:29.773418+0000 2024-03-25T18:00:29.773 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:00:29.773 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:29.773 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:00:29.773 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:00:29.773 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:00:29.773 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:29.774 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.772+0000 7f14c81b5640 1 Processor -- start 2024-03-25T18:00:29.774 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.773+0000 7f14c81b5640 1 -- start start 2024-03-25T18:00:29.774 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.773+0000 7f14c81b5640 10 monclient: init 2024-03-25T18:00:29.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.773+0000 7f14c81b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:00:29.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.773+0000 7f14c81b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:00:29.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.773+0000 7f14c81b5640 10 monclient(hunting): picked mon.noname-b con 0x7f14c00b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T18:00:29.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.773+0000 7f14c81b5640 10 monclient(hunting): picked mon.noname-c con 0x7f14c00b1730 addr v1:172.21.15.169:6789/0 2024-03-25T18:00:29.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.773+0000 7f14c81b5640 10 monclient(hunting): picked mon.noname-a con 0x7f14c014be90 addr v1:172.21.15.18:6789/0 2024-03-25T18:00:29.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.773+0000 7f14c81b5640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c00af940 con 0x7f14c014be90 2024-03-25T18:00:29.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.774+0000 7f14c81b5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c00b0a70 con 0x7f14c00b1730 2024-03-25T18:00:29.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.774+0000 7f14c81b5640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c00b0d50 con 0x7f14c00b4d60 2024-03-25T18:00:29.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.774+0000 7f14c81b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:00:29.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.774+0000 7f14c81b5640 10 monclient(hunting): authenticate will time out at 7158.352690s 2024-03-25T18:00:29.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.779+0000 7f14c5750640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f14c00b1730 0x7f14c00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:51502/0 (socket says 172.21.15.18:51502) 2024-03-25T18:00:29.780 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.779+0000 7f14c5750640 1 -- 172.21.15.18:0/2443403585 learned_addr learned my addr 172.21.15.18:0/2443403585 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:00:29.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.792+0000 7f14c4f4f640 1 -- 172.21.15.18:0/2443403585 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41790330 0 0) 0x7f14c00b0a70 con 0x7f14c00b1730 2024-03-25T18:00:29.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.792+0000 7f14c4f4f640 10 monclient(hunting): _negotiate 2024-03-25T18:00:29.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.792+0000 7f14c4f4f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:29.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.792+0000 7f14c4f4f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:29.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.792+0000 7f14c4f4f640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:00:29.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.792+0000 7f14c4f4f640 1 -- 172.21.15.18:0/2443403585 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14ac003520 con 0x7f14c00b1730 2024-03-25T18:00:29.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.801+0000 7f14c4f4f640 1 -- 172.21.15.18:0/2443403585 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798155690 0 0) 0x7f14c00af940 con 0x7f14c014be90 2024-03-25T18:00:29.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.801+0000 7f14c4f4f640 10 monclient(hunting): _negotiate 2024-03-25T18:00:29.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.801+0000 7f14c4f4f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:29.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.801+0000 7f14c4f4f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:29.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.801+0000 7f14c4f4f640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:00:29.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.801+0000 7f14c4f4f640 1 -- 172.21.15.18:0/2443403585 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c00b0a70 con 0x7f14c014be90 2024-03-25T18:00:29.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.802+0000 7f14c4f4f640 1 -- 172.21.15.18:0/2443403585 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429375316 0 0) 0x7f14c00b0d50 con 0x7f14c00b4d60 2024-03-25T18:00:29.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.802+0000 7f14c4f4f640 10 monclient(hunting): _negotiate 2024-03-25T18:00:29.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.802+0000 7f14c4f4f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:29.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.802+0000 7f14c4f4f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:29.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.802+0000 7f14c4f4f640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:00:29.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.802+0000 7f14c4f4f640 1 -- 172.21.15.18:0/2443403585 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c00af940 con 0x7f14c00b4d60 2024-03-25T18:00:29.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c4f4f640 1 -- 172.21.15.18:0/2443403585 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 497137348 0 0) 0x7f14ac003520 con 0x7f14c00b1730 2024-03-25T18:00:29.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c4f4f640 1 -- 172.21.15.18:0/2443403585 >> v1:172.21.15.18:6790/0 conn(0x7f14c00b4d60 legacy=0x7f14c013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:29.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c4f4f640 1 -- 172.21.15.18:0/2443403585 >> v1:172.21.15.18:6789/0 conn(0x7f14c014be90 legacy=0x7f14c014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:29.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c4f4f640 10 monclient: _finish_hunting 0 2024-03-25T18:00:29.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c4f4f640 1 monclient: found mon.noname-c 2024-03-25T18:00:29.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c4f4f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:00:29.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c4f4f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:00:29.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c4f4f640 1 -- 172.21.15.18:0/2443403585 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c00b1030 con 0x7f14c00b1730 2024-03-25T18:00:29.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c4f4f640 10 monclient: _finish_auth 0 2024-03-25T18:00:29.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c4f4f640 10 monclient: _check_auth_tickets 2024-03-25T18:00:29.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c4f4f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:00:29.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c4f4f640 1 -- 172.21.15.18:0/2443403585 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f14b0001690 con 0x7f14c00b1730 2024-03-25T18:00:29.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c81b5640 5 monclient: authenticate success, global_id 4396 2024-03-25T18:00:29.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c81b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:00:29.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c4f4f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:29.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c4f4f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:00:29.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c4f4f640 10 monclient: dump: 2024-03-25T18:00:29.810 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:29.810 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:29.810 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:29.810 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:29.810 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:29.810 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:29.810 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:29.810 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:29.810 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:29.810 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:29.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.807+0000 7f14c81b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:00:29.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.812+0000 7f14c4f4f640 1 -- 172.21.15.18:0/2443403585 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14b00017f0 con 0x7f14c00b1730 2024-03-25T18:00:29.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.812+0000 7f14c4f4f640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:00:29.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.812+0000 7f14c4f4f640 1 -- 172.21.15.18:0/2443403585 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f14b0001c00 con 0x7f14c00b1730 2024-03-25T18:00:29.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.812+0000 7f14c4f4f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:29.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.812+0000 7f14c4f4f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:00:29.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.812+0000 7f14c4f4f640 10 monclient: dump: 2024-03-25T18:00:29.813 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:29.813 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:29.813 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:29.813 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:29.813 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:29.813 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:29.813 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:29.814 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:29.814 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:29.814 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:29.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.812+0000 7f14c81b5640 10 monclient: get_monmap_and_config success 2024-03-25T18:00:29.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.812+0000 7f14c81b5640 4 set_mon_vals no callback set 2024-03-25T18:00:29.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.813+0000 7f14c81b5640 10 monclient: shutdown 2024-03-25T18:00:29.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.813+0000 7f14c81b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:00:29.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.813+0000 7f14c81b5640 1 -- 172.21.15.18:0/2443403585 >> v1:172.21.15.169:6789/0 conn(0x7f14c00b1730 legacy=0x7f14c00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:29.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.813+0000 7f14c81b5640 1 -- 172.21.15.18:0/2443403585 shutdown_connections 2024-03-25T18:00:29.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.813+0000 7f14c81b5640 1 -- 172.21.15.18:0/2443403585 >> 172.21.15.18:0/2443403585 conn(0x7f14c00617a0 msgr2=0x7f14c0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:00:29.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.813+0000 7f14c81b5640 1 -- 172.21.15.18:0/2443403585 shutdown_connections 2024-03-25T18:00:29.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.813+0000 7f14c81b5640 1 -- 172.21.15.18:0/2443403585 wait complete. 2024-03-25T18:00:29.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.814+0000 7f14c81b5640 10 monclient: build_initial_monmap 2024-03-25T18:00:29.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.814+0000 7f14c81b5640 10 monclient: monmap: 2024-03-25T18:00:29.817 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:00:29.817 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:00:29.817 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:00:29.817 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:00:29.817 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:00:29.817 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:00:29.817 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:00:29.817 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:00:29.817 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:00:29.817 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:29.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.814+0000 7f14c81b5640 1 librados: starting msgr at 2024-03-25T18:00:29.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.814+0000 7f14c81b5640 1 librados: starting objecter 2024-03-25T18:00:29.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.814+0000 7f14c81b5640 1 Processor -- start 2024-03-25T18:00:29.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.814+0000 7f14c81b5640 1 -- start start 2024-03-25T18:00:29.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.814+0000 7f14c81b5640 1 librados: setting wanted keys 2024-03-25T18:00:29.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.814+0000 7f14c81b5640 1 librados: calling monclient init 2024-03-25T18:00:29.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.814+0000 7f14c81b5640 10 monclient: init 2024-03-25T18:00:29.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.815+0000 7f14c81b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:00:29.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.815+0000 7f14c81b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:00:29.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.815+0000 7f14c81b5640 10 monclient(hunting): picked mon.noname-c con 0x7f14c00b1730 addr v1:172.21.15.18:6790/0 2024-03-25T18:00:29.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.815+0000 7f14c81b5640 10 monclient(hunting): picked mon.noname-a con 0x7f14c00b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T18:00:29.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.815+0000 7f14c81b5640 10 monclient(hunting): picked mon.noname-b con 0x7f14c014be90 addr v1:172.21.15.169:6789/0 2024-03-25T18:00:29.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.815+0000 7f14c81b5640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c01d83d0 con 0x7f14c00b4d60 2024-03-25T18:00:29.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.815+0000 7f14c81b5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c01d77a0 con 0x7f14c014be90 2024-03-25T18:00:29.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.815+0000 7f14c81b5640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c0138270 con 0x7f14c00b1730 2024-03-25T18:00:29.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.815+0000 7f14c81b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:00:29.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.815+0000 7f14c81b5640 10 monclient(hunting): authenticate will time out at 7158.393821s 2024-03-25T18:00:29.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.825+0000 7f14c5750640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f14c00b4d60 0x7f14c01405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:33632/0 (socket says 172.21.15.18:33632) 2024-03-25T18:00:29.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.825+0000 7f14c5750640 1 -- 172.21.15.18:0/3880904543 learned_addr learned my addr 172.21.15.18:0/3880904543 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:00:29.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.826+0000 7f14b67fc640 10 client.?.objecter ms_handle_connect 0x7f14c014be90 2024-03-25T18:00:29.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.826+0000 7f14b67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:29.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.826+0000 7f14b67fc640 10 client.?.objecter ms_handle_connect 0x7f14c00b4d60 2024-03-25T18:00:29.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.826+0000 7f14b67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:29.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.831+0000 7f14b67fc640 10 client.?.objecter ms_handle_connect 0x7f14c00b1730 2024-03-25T18:00:29.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.831+0000 7f14b67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:29.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.832+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451248526 0 0) 0x7f14c01d77a0 con 0x7f14c014be90 2024-03-25T18:00:29.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.832+0000 7f14b67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:00:29.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.832+0000 7f14b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:29.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.832+0000 7f14b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:29.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.832+0000 7f14b67fc640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:00:29.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.833+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14a4004eb0 con 0x7f14c014be90 2024-03-25T18:00:29.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.842+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315895452 0 0) 0x7f14c01d83d0 con 0x7f14c00b4d60 2024-03-25T18:00:29.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.842+0000 7f14b67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:00:29.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.842+0000 7f14b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:29.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.842+0000 7f14b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:29.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.842+0000 7f14b67fc640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:00:29.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.842+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c01d77a0 con 0x7f14c00b4d60 2024-03-25T18:00:29.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.845+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 717761468 0 0) 0x7f14a4004eb0 con 0x7f14c014be90 2024-03-25T18:00:29.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.845+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14c01d83d0 con 0x7f14c014be90 2024-03-25T18:00:29.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.845+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f14b8001bd0 con 0x7f14c014be90 2024-03-25T18:00:29.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.845+0000 7f14b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:29.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.845+0000 7f14b67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:00:29.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.845+0000 7f14b67fc640 10 monclient(hunting): dump: 2024-03-25T18:00:29.847 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:29.847 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:29.848 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:29.848 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:29.848 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:29.848 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:29.848 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:29.849 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:29.849 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:29.849 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:29.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.845+0000 7f14b67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:00:29.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.846+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183603722 0 0) 0x7f14c0138270 con 0x7f14c00b1730 2024-03-25T18:00:29.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.846+0000 7f14b67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:00:29.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.846+0000 7f14b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:29.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.846+0000 7f14b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:29.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.846+0000 7f14b67fc640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:00:29.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.846+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14a4004eb0 con 0x7f14c00b1730 2024-03-25T18:00:29.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3469250271 0 0) 0x7f14c01d83d0 con 0x7f14c014be90 2024-03-25T18:00:29.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 >> v1:172.21.15.18:6790/0 conn(0x7f14c00b1730 legacy=0x7f14c01d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:29.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 >> v1:172.21.15.18:6789/0 conn(0x7f14c00b4d60 legacy=0x7f14c01405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:29.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14b67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:00:29.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14b67fc640 1 monclient: found mon.b 2024-03-25T18:00:29.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:00:29.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14b67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:29.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c0140d00 con 0x7f14c014be90 2024-03-25T18:00:29.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14b67fc640 10 monclient: _finish_auth 0 2024-03-25T18:00:29.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14b67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:00:29.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:00:29.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14c81b5640 5 monclient: authenticate success, global_id 4399 2024-03-25T18:00:29.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14c81b5640 10 monclient: _renew_subs 2024-03-25T18:00:29.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14c81b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:29.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14c81b5640 1 -- 172.21.15.18:0/3880904543 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c0138a80 con 0x7f14c014be90 2024-03-25T18:00:29.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14c81b5640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:00:29.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14c81b5640 10 monclient: _renew_subs 2024-03-25T18:00:29.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14c81b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:29.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14c81b5640 1 -- 172.21.15.18:0/3880904543 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c0143640 con 0x7f14c014be90 2024-03-25T18:00:29.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.864+0000 7f14c81b5640 1 librados: init done 2024-03-25T18:00:29.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.866+0000 7f14c81b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:29.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.866+0000 7f14c81b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:29.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.866+0000 7f14c81b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:29.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:29.866+0000 7f14c81b5640 1 -- 172.21.15.18:0/3880904543 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c0140ef0 con 0x7f14c014be90 2024-03-25T18:00:30.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.162+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14b8003970 con 0x7f14c014be90 2024-03-25T18:00:30.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.162+0000 7f14b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:00:30.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.162+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f14b8003ae0 con 0x7f14c014be90 2024-03-25T18:00:30.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.162+0000 7f14b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:30.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.162+0000 7f14b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:00:30.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.162+0000 7f14b67fc640 10 monclient: dump: 2024-03-25T18:00:30.164 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:30.164 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:30.164 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:30.164 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:30.164 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:30.164 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:30.164 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:30.164 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:30.164 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:30.164 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:30.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.162+0000 7f14b7fff640 4 set_mon_vals no callback set 2024-03-25T18:00:30.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.163+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f14b8019ab0 con 0x7f14c014be90 2024-03-25T18:00:30.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.163+0000 7f14c5f51640 10 monclient: get_auth_request con 0x7f14a406de20 auth_method 0 2024-03-25T18:00:30.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.164+0000 7f14b67fc640 10 client.4399.objecter ms_handle_connect 0x7f14a406de20 2024-03-25T18:00:30.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.167+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f14b8053600 con 0x7f14c014be90 2024-03-25T18:00:30.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.167+0000 7f14b67fc640 10 client.4399.objecter ms_dispatch 0x7f14c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:00:30.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.167+0000 7f14b67fc640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:00:30.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.167+0000 7f14b67fc640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:00:30.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.167+0000 7f14b67fc640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-25T18:00:30.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.169+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f14b804ef40 con 0x7f14c014be90 2024-03-25T18:00:30.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.169+0000 7f14b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:30.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.169+0000 7f14b67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:00:30.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.421+0000 7f14c81b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:30.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.421+0000 7f14c81b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:30.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.421+0000 7f14c81b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:30.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.421+0000 7f14c81b5640 1 -- 172.21.15.18:0/3880904543 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14c00afbd0 con 0x7f14c014be90 2024-03-25T18:00:30.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.450+0000 7f14b67fc640 1 -- 172.21.15.18:0/3880904543 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f14b8051040 con 0x7f14c014be90 2024-03-25T18:00:30.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.450+0000 7f14b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:30.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.450+0000 7f14b67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:00:30.451 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:00:30.451 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:00:30.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.452+0000 7f14c81b5640 10 librados: watch_flush enter 2024-03-25T18:00:30.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.452+0000 7f14c81b5640 10 librados: watch_flush exit 2024-03-25T18:00:30.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.452+0000 7f14c81b5640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-25T18:00:30.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.452+0000 7f14c81b5640 10 client.4399.objecter successfully canceled tick 2024-03-25T18:00:30.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.452+0000 7f14c81b5640 1 -- 172.21.15.18:0/3880904543 >> v1:172.21.15.18:6800/3721244451 conn(0x7f14a406de20 legacy=0x7f14a40702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:30.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.452+0000 7f14c81b5640 10 monclient: shutdown 2024-03-25T18:00:30.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.452+0000 7f14c81b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:00:30.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.452+0000 7f14c81b5640 1 -- 172.21.15.18:0/3880904543 >> v1:172.21.15.169:6789/0 conn(0x7f14c014be90 legacy=0x7f14c0142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:30.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.452+0000 7f14c81b5640 1 -- 172.21.15.18:0/3880904543 shutdown_connections 2024-03-25T18:00:30.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.452+0000 7f14c81b5640 1 -- 172.21.15.18:0/3880904543 >> 172.21.15.18:0/3880904543 conn(0x7f14c00617a0 msgr2=0x7f14c0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:00:30.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.452+0000 7f14c81b5640 1 -- 172.21.15.18:0/3880904543 shutdown_connections 2024-03-25T18:00:30.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.452+0000 7f14c81b5640 1 -- 172.21.15.18:0/3880904543 wait complete. 2024-03-25T18:00:30.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:30.452+0000 7f14c81b5640 1 librados: shutdown 2024-03-25T18:00:30.464 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:00:37.271 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:00:37.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.387+0000 7f92d777b640 10 monclient: get_monmap_and_config 2024-03-25T18:00:37.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.387+0000 7f92d777b640 10 monclient: build_initial_monmap 2024-03-25T18:00:37.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.387+0000 7f92d777b640 10 monclient: monmap: 2024-03-25T18:00:37.388 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:00:37.388 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:37.388 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:00:37.388449+0000 2024-03-25T18:00:37.388 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:00:37.388449+0000 2024-03-25T18:00:37.388 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:00:37.388 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:37.388 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:00:37.388 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:00:37.388 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:00:37.389 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:37.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.387+0000 7f92d777b640 1 Processor -- start 2024-03-25T18:00:37.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.388+0000 7f92d777b640 1 -- start start 2024-03-25T18:00:37.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.388+0000 7f92d777b640 10 monclient: init 2024-03-25T18:00:37.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.388+0000 7f92d777b640 10 monclient: _reopen_session rank -1 2024-03-25T18:00:37.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.388+0000 7f92d777b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:00:37.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.388+0000 7f92d777b640 10 monclient(hunting): picked mon.noname-c con 0x7f92d00b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T18:00:37.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.388+0000 7f92d777b640 10 monclient(hunting): picked mon.noname-b con 0x7f92d00b1730 addr v1:172.21.15.18:6790/0 2024-03-25T18:00:37.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.388+0000 7f92d777b640 10 monclient(hunting): picked mon.noname-a con 0x7f92d014be90 addr v1:172.21.15.18:6789/0 2024-03-25T18:00:37.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.388+0000 7f92d777b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d00af940 con 0x7f92d014be90 2024-03-25T18:00:37.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.388+0000 7f92d777b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d00b0a70 con 0x7f92d00b4d60 2024-03-25T18:00:37.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.388+0000 7f92d777b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d00b0d50 con 0x7f92d00b1730 2024-03-25T18:00:37.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.388+0000 7f92d777b640 10 monclient(hunting): _renew_subs 2024-03-25T18:00:37.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.388+0000 7f92d777b640 10 monclient(hunting): authenticate will time out at 7165.967612s 2024-03-25T18:00:37.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.400+0000 7f92d5d18640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f92d014be90 0x7f92d014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:57620/0 (socket says 172.21.15.18:57620) 2024-03-25T18:00:37.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.400+0000 7f92d5d18640 1 -- 172.21.15.18:0/1761666884 learned_addr learned my addr 172.21.15.18:0/1761666884 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:00:37.411 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.411+0000 7f92c7fff640 1 -- 172.21.15.18:0/1761666884 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806500501 0 0) 0x7f92d00af940 con 0x7f92d014be90 2024-03-25T18:00:37.412 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.411+0000 7f92c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:00:37.412 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.411+0000 7f92c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:37.412 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.411+0000 7f92c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:37.412 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.411+0000 7f92c7fff640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:00:37.412 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.411+0000 7f92c7fff640 1 -- 172.21.15.18:0/1761666884 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92bc003520 con 0x7f92d014be90 2024-03-25T18:00:37.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.423+0000 7f92c7fff640 1 -- 172.21.15.18:0/1761666884 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948935164 0 0) 0x7f92d00b0a70 con 0x7f92d00b4d60 2024-03-25T18:00:37.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.423+0000 7f92c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:00:37.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.423+0000 7f92c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:37.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.423+0000 7f92c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:37.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.423+0000 7f92c7fff640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:00:37.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.423+0000 7f92c7fff640 1 -- 172.21.15.18:0/1761666884 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d00af940 con 0x7f92d00b4d60 2024-03-25T18:00:37.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.426+0000 7f92c7fff640 1 -- 172.21.15.18:0/1761666884 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025466438 0 0) 0x7f92d00b0d50 con 0x7f92d00b1730 2024-03-25T18:00:37.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.426+0000 7f92c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:00:37.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.426+0000 7f92c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:37.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.426+0000 7f92c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:37.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.426+0000 7f92c7fff640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:00:37.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.426+0000 7f92c7fff640 1 -- 172.21.15.18:0/1761666884 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d00b0a70 con 0x7f92d00b1730 2024-03-25T18:00:37.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.430+0000 7f92c7fff640 1 -- 172.21.15.18:0/1761666884 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 740106814 0 0) 0x7f92bc003520 con 0x7f92d014be90 2024-03-25T18:00:37.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.431+0000 7f92c7fff640 1 -- 172.21.15.18:0/1761666884 >> v1:172.21.15.18:6790/0 conn(0x7f92d00b1730 legacy=0x7f92d00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:37.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.431+0000 7f92c7fff640 1 -- 172.21.15.18:0/1761666884 >> v1:172.21.15.169:6789/0 conn(0x7f92d00b4d60 legacy=0x7f92d013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:37.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.431+0000 7f92c7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:00:37.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.431+0000 7f92c7fff640 1 monclient: found mon.noname-a 2024-03-25T18:00:37.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.431+0000 7f92c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:00:37.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.431+0000 7f92c7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:00:37.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.431+0000 7f92c7fff640 1 -- 172.21.15.18:0/1761666884 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d00b1030 con 0x7f92d014be90 2024-03-25T18:00:37.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.431+0000 7f92c7fff640 10 monclient: _finish_auth 0 2024-03-25T18:00:37.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.431+0000 7f92c7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:00:37.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.431+0000 7f92c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:00:37.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.431+0000 7f92c7fff640 1 -- 172.21.15.18:0/1761666884 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f92c8001a10 con 0x7f92d014be90 2024-03-25T18:00:37.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.431+0000 7f92d777b640 5 monclient: authenticate success, global_id 4413 2024-03-25T18:00:37.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.431+0000 7f92d777b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:00:37.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.431+0000 7f92c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:37.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.431+0000 7f92c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:00:37.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.431+0000 7f92c7fff640 10 monclient: dump: 2024-03-25T18:00:37.433 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:37.433 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:37.433 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:37.433 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:37.433 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:37.433 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:37.433 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:37.433 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:37.433 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:37.433 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:37.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.431+0000 7f92d777b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:00:37.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.458+0000 7f92c7fff640 1 -- 172.21.15.18:0/1761666884 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92c8001b70 con 0x7f92d014be90 2024-03-25T18:00:37.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.458+0000 7f92c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:00:37.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.458+0000 7f92c7fff640 1 -- 172.21.15.18:0/1761666884 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f92c8001f80 con 0x7f92d014be90 2024-03-25T18:00:37.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.458+0000 7f92d777b640 10 monclient: get_monmap_and_config success 2024-03-25T18:00:37.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.458+0000 7f92d777b640 4 set_mon_vals no callback set 2024-03-25T18:00:37.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.459+0000 7f92c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:37.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.459+0000 7f92c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:00:37.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.459+0000 7f92c7fff640 10 monclient: dump: 2024-03-25T18:00:37.460 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:37.460 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:37.460 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:37.460 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:37.460 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:37.460 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:37.460 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:37.460 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:37.461 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:37.461 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:37.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.459+0000 7f92d777b640 10 monclient: shutdown 2024-03-25T18:00:37.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.459+0000 7f92d777b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:00:37.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.459+0000 7f92d777b640 1 -- 172.21.15.18:0/1761666884 >> v1:172.21.15.18:6789/0 conn(0x7f92d014be90 legacy=0x7f92d014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:37.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.459+0000 7f92d777b640 1 -- 172.21.15.18:0/1761666884 shutdown_connections 2024-03-25T18:00:37.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.459+0000 7f92d777b640 1 -- 172.21.15.18:0/1761666884 >> 172.21.15.18:0/1761666884 conn(0x7f92d00617a0 msgr2=0x7f92d0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:00:37.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.460+0000 7f92d777b640 1 -- 172.21.15.18:0/1761666884 shutdown_connections 2024-03-25T18:00:37.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.460+0000 7f92d777b640 1 -- 172.21.15.18:0/1761666884 wait complete. 2024-03-25T18:00:37.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.460+0000 7f92d777b640 10 monclient: build_initial_monmap 2024-03-25T18:00:37.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.460+0000 7f92d777b640 10 monclient: monmap: 2024-03-25T18:00:37.463 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:00:37.463 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:00:37.463 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:00:37.463 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:00:37.463 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:00:37.463 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:00:37.463 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:00:37.463 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:00:37.463 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:00:37.463 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:37.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.461+0000 7f92d777b640 1 librados: starting msgr at 2024-03-25T18:00:37.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.461+0000 7f92d777b640 1 librados: starting objecter 2024-03-25T18:00:37.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.461+0000 7f92d777b640 1 Processor -- start 2024-03-25T18:00:37.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.461+0000 7f92d777b640 1 -- start start 2024-03-25T18:00:37.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.461+0000 7f92d777b640 1 librados: setting wanted keys 2024-03-25T18:00:37.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.461+0000 7f92d777b640 1 librados: calling monclient init 2024-03-25T18:00:37.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.461+0000 7f92d777b640 10 monclient: init 2024-03-25T18:00:37.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.462+0000 7f92d777b640 10 monclient: _reopen_session rank -1 2024-03-25T18:00:37.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.462+0000 7f92d777b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:00:37.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.462+0000 7f92d777b640 10 monclient(hunting): picked mon.noname-b con 0x7f92d00b1730 addr v1:172.21.15.169:6789/0 2024-03-25T18:00:37.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.462+0000 7f92d777b640 10 monclient(hunting): picked mon.noname-a con 0x7f92d00b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T18:00:37.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.462+0000 7f92d777b640 10 monclient(hunting): picked mon.noname-c con 0x7f92d014be90 addr v1:172.21.15.18:6790/0 2024-03-25T18:00:37.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.462+0000 7f92d777b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d01d86f0 con 0x7f92d00b4d60 2024-03-25T18:00:37.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.462+0000 7f92d777b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d013b4f0 con 0x7f92d00b1730 2024-03-25T18:00:37.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.462+0000 7f92d777b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d01d7eb0 con 0x7f92d014be90 2024-03-25T18:00:37.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.462+0000 7f92d777b640 10 monclient(hunting): _renew_subs 2024-03-25T18:00:37.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.462+0000 7f92d777b640 10 monclient(hunting): authenticate will time out at 7166.040958s 2024-03-25T18:00:37.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.471+0000 7f92d5517640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f92d00b1730 0x7f92d01d77a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:37430/0 (socket says 172.21.15.18:37430) 2024-03-25T18:00:37.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.471+0000 7f92d5517640 1 -- 172.21.15.18:0/2520032735 learned_addr learned my addr 172.21.15.18:0/2520032735 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:00:37.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.472+0000 7f92c5ffb640 10 client.?.objecter ms_handle_connect 0x7f92d00b1730 2024-03-25T18:00:37.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.472+0000 7f92c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:37.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.482+0000 7f92c5ffb640 10 client.?.objecter ms_handle_connect 0x7f92d00b4d60 2024-03-25T18:00:37.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.482+0000 7f92c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:37.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.486+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684161771 0 0) 0x7f92d013b4f0 con 0x7f92d00b1730 2024-03-25T18:00:37.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.486+0000 7f92c5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:00:37.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.486+0000 7f92c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:37.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.486+0000 7f92c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:37.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.486+0000 7f92c5ffb640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:00:37.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.486+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b4004eb0 con 0x7f92d00b1730 2024-03-25T18:00:37.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.488+0000 7f92c5ffb640 10 client.?.objecter ms_handle_connect 0x7f92d014be90 2024-03-25T18:00:37.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.488+0000 7f92c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:37.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.504+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947634021 0 0) 0x7f92d00b0a70 con 0x7f92d00b4d60 2024-03-25T18:00:37.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.504+0000 7f92c5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:00:37.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.504+0000 7f92c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:37.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.504+0000 7f92c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:37.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.504+0000 7f92c5ffb640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:00:37.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.504+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d013b4f0 con 0x7f92d00b4d60 2024-03-25T18:00:37.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.511+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 113039864 0 0) 0x7f92b4004eb0 con 0x7f92d00b1730 2024-03-25T18:00:37.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.511+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92d00b0a70 con 0x7f92d00b1730 2024-03-25T18:00:37.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.511+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f92cc001a30 con 0x7f92d00b1730 2024-03-25T18:00:37.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.511+0000 7f92c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:37.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.511+0000 7f92c5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:00:37.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.511+0000 7f92c5ffb640 10 monclient(hunting): dump: 2024-03-25T18:00:37.513 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:37.513 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:37.513 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:37.513 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:37.513 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:37.513 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:37.513 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:37.513 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:37.513 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:37.513 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:37.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.511+0000 7f92c5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:00:37.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.517+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404857844 0 0) 0x7f92d01d7eb0 con 0x7f92d014be90 2024-03-25T18:00:37.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.517+0000 7f92c5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:00:37.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.517+0000 7f92c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:37.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.517+0000 7f92c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:37.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.517+0000 7f92c5ffb640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:00:37.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.517+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b4004eb0 con 0x7f92d014be90 2024-03-25T18:00:37.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.522+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3510467817 0 0) 0x7f92d013b4f0 con 0x7f92d00b4d60 2024-03-25T18:00:37.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.522+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92d01d7eb0 con 0x7f92d00b4d60 2024-03-25T18:00:37.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.522+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f92c00016d0 con 0x7f92d00b4d60 2024-03-25T18:00:37.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.522+0000 7f92c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:37.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.522+0000 7f92c5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:00:37.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.522+0000 7f92c5ffb640 10 monclient(hunting): dump: 2024-03-25T18:00:37.523 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:37.523 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:37.524 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:37.524 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:37.524 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:37.524 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:37.524 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:37.524 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:37.524 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:37.525 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:37.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2010268423 0 0) 0x7f92d00b0a70 con 0x7f92d00b1730 2024-03-25T18:00:37.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 >> v1:172.21.15.18:6790/0 conn(0x7f92d014be90 legacy=0x7f92d01e3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:37.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 >> v1:172.21.15.18:6789/0 conn(0x7f92d00b4d60 legacy=0x7f92d0136fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:37.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92c5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:00:37.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92c5ffb640 1 monclient: found mon.b 2024-03-25T18:00:37.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:00:37.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92c5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:37.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d01d8040 con 0x7f92d00b1730 2024-03-25T18:00:37.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92c5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:00:37.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92c5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:00:37.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:00:37.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92d777b640 5 monclient: authenticate success, global_id 4405 2024-03-25T18:00:37.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92d777b640 10 monclient: _renew_subs 2024-03-25T18:00:37.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92d777b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:37.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92d777b640 1 -- 172.21.15.18:0/2520032735 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d013ba70 con 0x7f92d00b1730 2024-03-25T18:00:37.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92d777b640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:00:37.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92d777b640 10 monclient: _renew_subs 2024-03-25T18:00:37.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92d777b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:37.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92d777b640 1 -- 172.21.15.18:0/2520032735 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d01378d0 con 0x7f92d00b1730 2024-03-25T18:00:37.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.531+0000 7f92d777b640 1 librados: init done 2024-03-25T18:00:37.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.533+0000 7f92d777b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:37.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.533+0000 7f92d777b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:37.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.533+0000 7f92d777b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:37.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.533+0000 7f92d777b640 1 -- 172.21.15.18:0/2520032735 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d00627b0 con 0x7f92d00b1730 2024-03-25T18:00:37.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.840+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92cc001c70 con 0x7f92d00b1730 2024-03-25T18:00:37.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.840+0000 7f92c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:00:37.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.840+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f92cc003a60 con 0x7f92d00b1730 2024-03-25T18:00:37.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.840+0000 7f92c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:37.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.840+0000 7f92c5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:00:37.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.840+0000 7f92c5ffb640 10 monclient: dump: 2024-03-25T18:00:37.842 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:37.842 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:37.842 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:37.842 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:37.842 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:37.842 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:37.842 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:37.842 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:37.842 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:37.842 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:37.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.840+0000 7f92c77fe640 4 set_mon_vals no callback set 2024-03-25T18:00:37.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.841+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f92cc003c80 con 0x7f92d00b1730 2024-03-25T18:00:37.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.842+0000 7f92d4d16640 10 monclient: get_auth_request con 0x7f92b406dfb0 auth_method 0 2024-03-25T18:00:37.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.842+0000 7f92c5ffb640 10 client.4405.objecter ms_handle_connect 0x7f92b406dfb0 2024-03-25T18:00:37.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.846+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f92cc0548a0 con 0x7f92d00b1730 2024-03-25T18:00:37.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.846+0000 7f92c5ffb640 10 client.4405.objecter ms_dispatch 0x7f92d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:00:37.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.846+0000 7f92c5ffb640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:00:37.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.846+0000 7f92c5ffb640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:00:37.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.846+0000 7f92c5ffb640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-25T18:00:37.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.847+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92cc0503b0 con 0x7f92d00b1730 2024-03-25T18:00:37.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.847+0000 7f92c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:37.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:37.847+0000 7f92c5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:00:38.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.106+0000 7f92d777b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:38.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.106+0000 7f92d777b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:38.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.106+0000 7f92d777b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:38.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.106+0000 7f92d777b640 1 -- 172.21.15.18:0/2520032735 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92d0069f40 con 0x7f92d00b1730 2024-03-25T18:00:38.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.181+0000 7f92c5ffb640 1 -- 172.21.15.18:0/2520032735 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f92cc0522e0 con 0x7f92d00b1730 2024-03-25T18:00:38.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.181+0000 7f92c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:38.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.181+0000 7f92c5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:00:38.183 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:00:38.183 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:00:38.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.183+0000 7f92d777b640 10 librados: watch_flush enter 2024-03-25T18:00:38.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.183+0000 7f92d777b640 10 librados: watch_flush exit 2024-03-25T18:00:38.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.183+0000 7f92d777b640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-25T18:00:38.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.183+0000 7f92d777b640 10 client.4405.objecter successfully canceled tick 2024-03-25T18:00:38.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.184+0000 7f92d777b640 1 -- 172.21.15.18:0/2520032735 >> v1:172.21.15.18:6800/3721244451 conn(0x7f92b406dfb0 legacy=0x7f92b4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:38.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.184+0000 7f92d777b640 10 monclient: shutdown 2024-03-25T18:00:38.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.184+0000 7f92d777b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:00:38.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.184+0000 7f92d777b640 1 -- 172.21.15.18:0/2520032735 >> v1:172.21.15.169:6789/0 conn(0x7f92d00b1730 legacy=0x7f92d01d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:38.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.184+0000 7f92d777b640 1 -- 172.21.15.18:0/2520032735 shutdown_connections 2024-03-25T18:00:38.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.184+0000 7f92d777b640 1 -- 172.21.15.18:0/2520032735 >> 172.21.15.18:0/2520032735 conn(0x7f92d00617a0 msgr2=0x7f92d013c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:00:38.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.184+0000 7f92d777b640 1 -- 172.21.15.18:0/2520032735 shutdown_connections 2024-03-25T18:00:38.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.184+0000 7f92d777b640 1 -- 172.21.15.18:0/2520032735 wait complete. 2024-03-25T18:00:38.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:38.184+0000 7f92d777b640 1 librados: shutdown 2024-03-25T18:00:38.197 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:00:42.809 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:04:40.602 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:42.817 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:04:40.610 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:42.860 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:04:40.653 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:42.872 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:04:40.621 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:42.874 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:04:40.667 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:42.877 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:04:40.627 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:42.880 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:04:40.673 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:42.905 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:04:40.654 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:42.910 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:04:40.703 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:00:45.004 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:00:45.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.106+0000 7f621f989640 10 monclient: get_monmap_and_config 2024-03-25T18:00:45.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.106+0000 7f621f989640 10 monclient: build_initial_monmap 2024-03-25T18:00:45.107 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.106+0000 7f621f989640 10 monclient: monmap: 2024-03-25T18:00:45.107 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:00:45.107 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:45.107 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:00:45.107951+0000 2024-03-25T18:00:45.107 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:00:45.107951+0000 2024-03-25T18:00:45.107 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:00:45.108 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:45.108 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:00:45.108 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:00:45.108 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:00:45.108 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:45.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.107+0000 7f621f989640 1 Processor -- start 2024-03-25T18:00:45.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.107+0000 7f621f989640 1 -- start start 2024-03-25T18:00:45.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.107+0000 7f621f989640 10 monclient: init 2024-03-25T18:00:45.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.107+0000 7f621f989640 10 monclient: _reopen_session rank -1 2024-03-25T18:00:45.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.107+0000 7f621f989640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:00:45.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.107+0000 7f621f989640 10 monclient(hunting): picked mon.noname-a con 0x7f62180b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T18:00:45.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.107+0000 7f621f989640 10 monclient(hunting): picked mon.noname-b con 0x7f62180b1730 addr v1:172.21.15.18:6790/0 2024-03-25T18:00:45.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.107+0000 7f621f989640 10 monclient(hunting): picked mon.noname-c con 0x7f621814be90 addr v1:172.21.15.169:6789/0 2024-03-25T18:00:45.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.107+0000 7f621f989640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62180af940 con 0x7f62180b4d60 2024-03-25T18:00:45.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.107+0000 7f621f989640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62180b0a70 con 0x7f621814be90 2024-03-25T18:00:45.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.107+0000 7f621f989640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62180b0d50 con 0x7f62180b1730 2024-03-25T18:00:45.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.108+0000 7f621f989640 10 monclient(hunting): _renew_subs 2024-03-25T18:00:45.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.108+0000 7f621f989640 10 monclient(hunting): authenticate will time out at 7173.686665s 2024-03-25T18:00:45.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.108+0000 7f621cf24640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f62180b1730 0x7f62180b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:37632/0 (socket says 172.21.15.18:37632) 2024-03-25T18:00:45.110 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.108+0000 7f621cf24640 1 -- 172.21.15.18:0/1959122906 learned_addr learned my addr 172.21.15.18:0/1959122906 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:00:45.139 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.138+0000 7f620ffff640 1 -- 172.21.15.18:0/1959122906 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253444860 0 0) 0x7f62180af940 con 0x7f62180b4d60 2024-03-25T18:00:45.139 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.138+0000 7f620ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:00:45.139 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.138+0000 7f620ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:45.139 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.138+0000 7f620ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:45.139 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.138+0000 7f620ffff640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:00:45.139 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.138+0000 7f620ffff640 1 -- 172.21.15.18:0/1959122906 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6204003520 con 0x7f62180b4d60 2024-03-25T18:00:45.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 1 -- 172.21.15.18:0/1959122906 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643369102 0 0) 0x7f62180b0a70 con 0x7f621814be90 2024-03-25T18:00:45.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:00:45.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:45.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:45.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:00:45.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 1 -- 172.21.15.18:0/1959122906 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62180af940 con 0x7f621814be90 2024-03-25T18:00:45.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 1 -- 172.21.15.18:0/1959122906 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2888633569 0 0) 0x7f6204003520 con 0x7f62180b4d60 2024-03-25T18:00:45.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 1 -- 172.21.15.18:0/1959122906 >> v1:172.21.15.18:6790/0 conn(0x7f62180b1730 legacy=0x7f62180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:45.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 1 -- 172.21.15.18:0/1959122906 >> v1:172.21.15.169:6789/0 conn(0x7f621814be90 legacy=0x7f621814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:45.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:00:45.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 1 monclient: found mon.noname-a 2024-03-25T18:00:45.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:00:45.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:00:45.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 1 -- 172.21.15.18:0/1959122906 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62180b1030 con 0x7f62180b4d60 2024-03-25T18:00:45.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 10 monclient: _finish_auth 0 2024-03-25T18:00:45.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:00:45.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:00:45.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 1 -- 172.21.15.18:0/1959122906 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f6214001690 con 0x7f62180b4d60 2024-03-25T18:00:45.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f621f989640 5 monclient: authenticate success, global_id 4419 2024-03-25T18:00:45.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f621f989640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:00:45.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:45.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:00:45.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f620ffff640 10 monclient: dump: 2024-03-25T18:00:45.183 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:45.183 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:45.183 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:45.183 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:45.183 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:45.183 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:45.184 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:45.184 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:45.184 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:45.184 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:45.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.180+0000 7f621f989640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:00:45.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.202+0000 7f620ffff640 1 -- 172.21.15.18:0/1959122906 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62140017f0 con 0x7f62180b4d60 2024-03-25T18:00:45.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.202+0000 7f620ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:00:45.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.202+0000 7f620ffff640 1 -- 172.21.15.18:0/1959122906 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f6214001c00 con 0x7f62180b4d60 2024-03-25T18:00:45.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.202+0000 7f620ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:45.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.202+0000 7f620ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:00:45.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.202+0000 7f620ffff640 10 monclient: dump: 2024-03-25T18:00:45.203 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:45.203 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:45.203 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:45.203 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:45.204 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:45.204 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:45.204 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:45.204 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:45.204 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:45.204 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:45.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.202+0000 7f621f989640 10 monclient: get_monmap_and_config success 2024-03-25T18:00:45.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.202+0000 7f621f989640 4 set_mon_vals no callback set 2024-03-25T18:00:45.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.203+0000 7f621f989640 10 monclient: shutdown 2024-03-25T18:00:45.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.203+0000 7f621f989640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:00:45.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.203+0000 7f621f989640 1 -- 172.21.15.18:0/1959122906 >> v1:172.21.15.18:6789/0 conn(0x7f62180b4d60 legacy=0x7f621813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:45.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.203+0000 7f621f989640 1 -- 172.21.15.18:0/1959122906 shutdown_connections 2024-03-25T18:00:45.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.203+0000 7f621f989640 1 -- 172.21.15.18:0/1959122906 >> 172.21.15.18:0/1959122906 conn(0x7f62180617a0 msgr2=0x7f6218061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:00:45.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.203+0000 7f621f989640 1 -- 172.21.15.18:0/1959122906 shutdown_connections 2024-03-25T18:00:45.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.203+0000 7f621f989640 1 -- 172.21.15.18:0/1959122906 wait complete. 2024-03-25T18:00:45.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.204+0000 7f621f989640 10 monclient: build_initial_monmap 2024-03-25T18:00:45.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.204+0000 7f621f989640 10 monclient: monmap: 2024-03-25T18:00:45.206 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:00:45.206 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:00:45.206 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:00:45.206 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:00:45.206 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:00:45.206 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:00:45.206 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:00:45.206 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:00:45.207 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:00:45.207 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:45.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.204+0000 7f621f989640 1 librados: starting msgr at 2024-03-25T18:00:45.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.204+0000 7f621f989640 1 librados: starting objecter 2024-03-25T18:00:45.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.204+0000 7f621f989640 1 Processor -- start 2024-03-25T18:00:45.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.204+0000 7f621f989640 1 -- start start 2024-03-25T18:00:45.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.204+0000 7f621f989640 1 librados: setting wanted keys 2024-03-25T18:00:45.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.204+0000 7f621f989640 1 librados: calling monclient init 2024-03-25T18:00:45.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.204+0000 7f621f989640 10 monclient: init 2024-03-25T18:00:45.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.205+0000 7f621f989640 10 monclient: _reopen_session rank -1 2024-03-25T18:00:45.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.205+0000 7f621f989640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:00:45.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.205+0000 7f621f989640 10 monclient(hunting): picked mon.noname-a con 0x7f62180b1730 addr v1:172.21.15.18:6789/0 2024-03-25T18:00:45.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.205+0000 7f621f989640 10 monclient(hunting): picked mon.noname-c con 0x7f62180b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T18:00:45.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.205+0000 7f621f989640 10 monclient(hunting): picked mon.noname-b con 0x7f621814be90 addr v1:172.21.15.169:6789/0 2024-03-25T18:00:45.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.205+0000 7f621f989640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6218144d40 con 0x7f62180b1730 2024-03-25T18:00:45.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.205+0000 7f621f989640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62181dbac0 con 0x7f621814be90 2024-03-25T18:00:45.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.205+0000 7f621f989640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6218135d30 con 0x7f62180b4d60 2024-03-25T18:00:45.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.205+0000 7f621f989640 10 monclient(hunting): _renew_subs 2024-03-25T18:00:45.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.205+0000 7f621f989640 10 monclient(hunting): authenticate will time out at 7173.784311s 2024-03-25T18:00:45.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.213+0000 7f621df26640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f621814be90 0x7f62181e1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:48966/0 (socket says 172.21.15.18:48966) 2024-03-25T18:00:45.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.213+0000 7f621df26640 1 -- 172.21.15.18:0/4081220663 learned_addr learned my addr 172.21.15.18:0/4081220663 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:00:45.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.213+0000 7f620dffb640 10 client.?.objecter ms_handle_connect 0x7f621814be90 2024-03-25T18:00:45.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.213+0000 7f620dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:45.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.215+0000 7f620dffb640 10 client.?.objecter ms_handle_connect 0x7f62180b4d60 2024-03-25T18:00:45.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.215+0000 7f620dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:45.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.217+0000 7f620dffb640 10 client.?.objecter ms_handle_connect 0x7f62180b1730 2024-03-25T18:00:45.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.217+0000 7f620dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:45.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.221+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121788749 0 0) 0x7f62181dbac0 con 0x7f621814be90 2024-03-25T18:00:45.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.221+0000 7f620dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:00:45.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.221+0000 7f620dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:45.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.221+0000 7f620dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:45.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.221+0000 7f620dffb640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:00:45.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.221+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61fc004eb0 con 0x7f621814be90 2024-03-25T18:00:45.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.237+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2084651146 0 0) 0x7f61fc004eb0 con 0x7f621814be90 2024-03-25T18:00:45.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.237+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62181dbac0 con 0x7f621814be90 2024-03-25T18:00:45.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.237+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f6210001c90 con 0x7f621814be90 2024-03-25T18:00:45.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.237+0000 7f620dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:45.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.237+0000 7f620dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:00:45.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.237+0000 7f620dffb640 10 monclient(hunting): dump: 2024-03-25T18:00:45.239 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:45.239 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:45.239 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:45.239 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:45.239 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:45.239 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:45.239 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:45.240 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:45.240 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:45.240 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:45.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.237+0000 7f620dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:00:45.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.239+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328779538 0 0) 0x7f6218144d40 con 0x7f62180b1730 2024-03-25T18:00:45.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.239+0000 7f620dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:00:45.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.239+0000 7f620dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:45.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.239+0000 7f620dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:45.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.239+0000 7f620dffb640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:00:45.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.239+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61fc004eb0 con 0x7f62180b1730 2024-03-25T18:00:45.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.243+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809497762 0 0) 0x7f6218135d30 con 0x7f62180b4d60 2024-03-25T18:00:45.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.243+0000 7f620dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:00:45.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.243+0000 7f620dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:45.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.243+0000 7f620dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:45.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.243+0000 7f620dffb640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:00:45.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.243+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6218144d40 con 0x7f62180b4d60 2024-03-25T18:00:45.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2317657791 0 0) 0x7f62181dbac0 con 0x7f621814be90 2024-03-25T18:00:45.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 >> v1:172.21.15.18:6790/0 conn(0x7f62180b4d60 legacy=0x7f6218144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:45.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 >> v1:172.21.15.18:6789/0 conn(0x7f62180b1730 legacy=0x7f62181db3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:45.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f620dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:00:45.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f620dffb640 1 monclient: found mon.b 2024-03-25T18:00:45.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f620dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:00:45.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f620dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:45.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62181382f0 con 0x7f621814be90 2024-03-25T18:00:45.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f620dffb640 10 monclient: _finish_auth 0 2024-03-25T18:00:45.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f620dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:00:45.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f620dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:00:45.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f621f989640 5 monclient: authenticate success, global_id 4411 2024-03-25T18:00:45.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f621f989640 10 monclient: _renew_subs 2024-03-25T18:00:45.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f621f989640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:45.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f621f989640 1 -- 172.21.15.18:0/4081220663 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6218145670 con 0x7f621814be90 2024-03-25T18:00:45.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f621f989640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:00:45.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f621f989640 10 monclient: _renew_subs 2024-03-25T18:00:45.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f621f989640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:45.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f621f989640 1 -- 172.21.15.18:0/4081220663 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6218145920 con 0x7f621814be90 2024-03-25T18:00:45.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.255+0000 7f621f989640 1 librados: init done 2024-03-25T18:00:45.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.256+0000 7f621f989640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:45.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.257+0000 7f621f989640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:45.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.257+0000 7f621f989640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:45.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.257+0000 7f621f989640 1 -- 172.21.15.18:0/4081220663 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62181448e0 con 0x7f621814be90 2024-03-25T18:00:45.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.536+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6210003990 con 0x7f621814be90 2024-03-25T18:00:45.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.536+0000 7f620dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:00:45.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.536+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f6210003b80 con 0x7f621814be90 2024-03-25T18:00:45.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.536+0000 7f620dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:45.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.536+0000 7f620dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:00:45.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.536+0000 7f620dffb640 10 monclient: dump: 2024-03-25T18:00:45.538 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:45.538 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:45.538 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:45.538 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:45.539 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:45.539 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:45.539 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:45.539 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:45.539 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:45.539 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:45.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.536+0000 7f620f7fe640 4 set_mon_vals no callback set 2024-03-25T18:00:45.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.537+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f6210003da0 con 0x7f621814be90 2024-03-25T18:00:45.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.538+0000 7f621d725640 10 monclient: get_auth_request con 0x7f61fc06dea0 auth_method 0 2024-03-25T18:00:45.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.538+0000 7f620dffb640 10 client.4411.objecter ms_handle_connect 0x7f61fc06dea0 2024-03-25T18:00:45.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.542+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f6210054890 con 0x7f621814be90 2024-03-25T18:00:45.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.542+0000 7f620dffb640 10 client.4411.objecter ms_dispatch 0x7f6218000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:00:45.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.542+0000 7f620dffb640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:00:45.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.542+0000 7f620dffb640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:00:45.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.542+0000 7f620dffb640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-25T18:00:45.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.542+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62100503a0 con 0x7f621814be90 2024-03-25T18:00:45.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.542+0000 7f620dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:45.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.542+0000 7f620dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:00:45.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.799+0000 7f621f989640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:45.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.799+0000 7f621f989640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:45.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.799+0000 7f621f989640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:45.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.799+0000 7f621f989640 1 -- 172.21.15.18:0/4081220663 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62180627b0 con 0x7f621814be90 2024-03-25T18:00:45.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.827+0000 7f620dffb640 1 -- 172.21.15.18:0/4081220663 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f62100522d0 con 0x7f621814be90 2024-03-25T18:00:45.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.827+0000 7f620dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:45.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.827+0000 7f620dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:00:45.829 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:00:45.829 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:00:45.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.830+0000 7f621f989640 10 librados: watch_flush enter 2024-03-25T18:00:45.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.830+0000 7f621f989640 10 librados: watch_flush exit 2024-03-25T18:00:45.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.830+0000 7f621f989640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-25T18:00:45.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.830+0000 7f621f989640 10 client.4411.objecter successfully canceled tick 2024-03-25T18:00:45.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.830+0000 7f621f989640 1 -- 172.21.15.18:0/4081220663 >> v1:172.21.15.18:6800/3721244451 conn(0x7f61fc06dea0 legacy=0x7f61fc070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:45.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.830+0000 7f621f989640 10 monclient: shutdown 2024-03-25T18:00:45.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.830+0000 7f621f989640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:00:45.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.830+0000 7f621f989640 1 -- 172.21.15.18:0/4081220663 >> v1:172.21.15.169:6789/0 conn(0x7f621814be90 legacy=0x7f62181e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:45.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.830+0000 7f621f989640 1 -- 172.21.15.18:0/4081220663 shutdown_connections 2024-03-25T18:00:45.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.830+0000 7f621f989640 1 -- 172.21.15.18:0/4081220663 >> 172.21.15.18:0/4081220663 conn(0x7f62180617a0 msgr2=0x7f62180b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:00:45.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.830+0000 7f621f989640 1 -- 172.21.15.18:0/4081220663 shutdown_connections 2024-03-25T18:00:45.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.830+0000 7f621f989640 1 -- 172.21.15.18:0/4081220663 wait complete. 2024-03-25T18:00:45.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:45.830+0000 7f621f989640 1 librados: shutdown 2024-03-25T18:00:45.842 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:00:52.649 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:00:52.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.759+0000 7fdcf7016640 10 monclient: get_monmap_and_config 2024-03-25T18:00:52.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.759+0000 7fdcf7016640 10 monclient: build_initial_monmap 2024-03-25T18:00:52.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.759+0000 7fdcf7016640 10 monclient: monmap: 2024-03-25T18:00:52.761 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:00:52.761 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:52.761 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:00:52.761258+0000 2024-03-25T18:00:52.761 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:00:52.761258+0000 2024-03-25T18:00:52.761 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:00:52.761 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:52.761 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:00:52.761 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:00:52.761 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:00:52.761 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:52.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.760+0000 7fdcf7016640 1 Processor -- start 2024-03-25T18:00:52.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.760+0000 7fdcf7016640 1 -- start start 2024-03-25T18:00:52.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.760+0000 7fdcf7016640 10 monclient: init 2024-03-25T18:00:52.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.761+0000 7fdcf7016640 10 monclient: _reopen_session rank -1 2024-03-25T18:00:52.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.761+0000 7fdcf7016640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:00:52.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.761+0000 7fdcf7016640 10 monclient(hunting): picked mon.noname-a con 0x7fdcf00b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T18:00:52.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.761+0000 7fdcf7016640 10 monclient(hunting): picked mon.noname-c con 0x7fdcf00b1730 addr v1:172.21.15.169:6789/0 2024-03-25T18:00:52.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.761+0000 7fdcf7016640 10 monclient(hunting): picked mon.noname-b con 0x7fdcf014be90 addr v1:172.21.15.18:6790/0 2024-03-25T18:00:52.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.761+0000 7fdcf7016640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf00af940 con 0x7fdcf00b4d60 2024-03-25T18:00:52.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.761+0000 7fdcf7016640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf00b0a70 con 0x7fdcf00b1730 2024-03-25T18:00:52.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.761+0000 7fdcf7016640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf00b0d50 con 0x7fdcf014be90 2024-03-25T18:00:52.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.761+0000 7fdcf7016640 10 monclient(hunting): _renew_subs 2024-03-25T18:00:52.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.761+0000 7fdcf7016640 10 monclient(hunting): authenticate will time out at 7181.340295s 2024-03-25T18:00:52.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.774+0000 7fdceffff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fdcf00b1730 0x7fdcf00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:53764/0 (socket says 172.21.15.18:53764) 2024-03-25T18:00:52.775 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.774+0000 7fdceffff640 1 -- 172.21.15.18:0/3742843708 learned_addr learned my addr 172.21.15.18:0/3742843708 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:00:52.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.796+0000 7fdcef7fe640 1 -- 172.21.15.18:0/3742843708 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216439939 0 0) 0x7fdcf00b0a70 con 0x7fdcf00b1730 2024-03-25T18:00:52.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.796+0000 7fdcef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:00:52.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.796+0000 7fdcef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:52.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.796+0000 7fdcef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:52.798 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.796+0000 7fdcef7fe640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:00:52.798 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.796+0000 7fdcef7fe640 1 -- 172.21.15.18:0/3742843708 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce0003520 con 0x7fdcf00b1730 2024-03-25T18:00:52.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.807+0000 7fdcef7fe640 1 -- 172.21.15.18:0/3742843708 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439126835 0 0) 0x7fdcf00b0d50 con 0x7fdcf014be90 2024-03-25T18:00:52.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.807+0000 7fdcef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:00:52.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.807+0000 7fdcef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:52.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.807+0000 7fdcef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:52.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.807+0000 7fdcef7fe640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:00:52.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.808+0000 7fdcef7fe640 1 -- 172.21.15.18:0/3742843708 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcf00b0a70 con 0x7fdcf014be90 2024-03-25T18:00:52.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcef7fe640 1 -- 172.21.15.18:0/3742843708 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4350618 0 0) 0x7fdce0003520 con 0x7fdcf00b1730 2024-03-25T18:00:52.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcef7fe640 1 -- 172.21.15.18:0/3742843708 >> v1:172.21.15.18:6790/0 conn(0x7fdcf014be90 legacy=0x7fdcf014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:52.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcef7fe640 1 -- 172.21.15.18:0/3742843708 >> v1:172.21.15.18:6789/0 conn(0x7fdcf00b4d60 legacy=0x7fdcf013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:52.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcef7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:00:52.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcef7fe640 1 monclient: found mon.noname-c 2024-03-25T18:00:52.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:00:52.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcef7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:00:52.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcef7fe640 1 -- 172.21.15.18:0/3742843708 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf00b1030 con 0x7fdcf00b1730 2024-03-25T18:00:52.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcef7fe640 10 monclient: _finish_auth 0 2024-03-25T18:00:52.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcef7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:00:52.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:00:52.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcef7fe640 1 -- 172.21.15.18:0/3742843708 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fdcdc001690 con 0x7fdcf00b1730 2024-03-25T18:00:52.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcf7016640 5 monclient: authenticate success, global_id 4414 2024-03-25T18:00:52.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcf7016640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:00:52.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:52.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:00:52.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcef7fe640 10 monclient: dump: 2024-03-25T18:00:52.811 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:52.811 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:52.811 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:52.811 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:52.811 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:52.811 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:52.812 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:52.812 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:52.812 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:52.812 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:52.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.809+0000 7fdcf7016640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:00:52.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.817+0000 7fdcef7fe640 1 -- 172.21.15.18:0/3742843708 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcdc0017f0 con 0x7fdcf00b1730 2024-03-25T18:00:52.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.817+0000 7fdcef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:00:52.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.817+0000 7fdcef7fe640 1 -- 172.21.15.18:0/3742843708 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fdcdc001c00 con 0x7fdcf00b1730 2024-03-25T18:00:52.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.817+0000 7fdcf7016640 10 monclient: get_monmap_and_config success 2024-03-25T18:00:52.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.817+0000 7fdcf7016640 4 set_mon_vals no callback set 2024-03-25T18:00:52.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.817+0000 7fdcef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:52.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.817+0000 7fdcef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:00:52.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.817+0000 7fdcef7fe640 10 monclient: dump: 2024-03-25T18:00:52.818 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:52.818 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:52.818 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:52.818 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:52.818 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:52.819 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:52.819 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:52.819 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:52.819 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:52.819 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:52.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.818+0000 7fdcf7016640 10 monclient: shutdown 2024-03-25T18:00:52.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.818+0000 7fdcf7016640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:00:52.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.818+0000 7fdcf7016640 1 -- 172.21.15.18:0/3742843708 >> v1:172.21.15.169:6789/0 conn(0x7fdcf00b1730 legacy=0x7fdcf00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:52.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.818+0000 7fdcf7016640 1 -- 172.21.15.18:0/3742843708 shutdown_connections 2024-03-25T18:00:52.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.818+0000 7fdcf7016640 1 -- 172.21.15.18:0/3742843708 >> 172.21.15.18:0/3742843708 conn(0x7fdcf00617a0 msgr2=0x7fdcf0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:00:52.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.818+0000 7fdcf7016640 1 -- 172.21.15.18:0/3742843708 shutdown_connections 2024-03-25T18:00:52.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.818+0000 7fdcf7016640 1 -- 172.21.15.18:0/3742843708 wait complete. 2024-03-25T18:00:52.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.818+0000 7fdcf7016640 10 monclient: build_initial_monmap 2024-03-25T18:00:52.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.818+0000 7fdcf7016640 10 monclient: monmap: 2024-03-25T18:00:52.821 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:00:52.821 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:00:52.821 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:00:52.822 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:00:52.822 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:00:52.822 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:00:52.822 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:00:52.822 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:00:52.822 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:00:52.822 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:52.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.819+0000 7fdcf7016640 1 librados: starting msgr at 2024-03-25T18:00:52.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.819+0000 7fdcf7016640 1 librados: starting objecter 2024-03-25T18:00:52.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.819+0000 7fdcf7016640 1 Processor -- start 2024-03-25T18:00:52.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.819+0000 7fdcf7016640 1 -- start start 2024-03-25T18:00:52.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.819+0000 7fdcf7016640 1 librados: setting wanted keys 2024-03-25T18:00:52.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.819+0000 7fdcf7016640 1 librados: calling monclient init 2024-03-25T18:00:52.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.819+0000 7fdcf7016640 10 monclient: init 2024-03-25T18:00:52.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.819+0000 7fdcf7016640 10 monclient: _reopen_session rank -1 2024-03-25T18:00:52.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.819+0000 7fdcf7016640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:00:52.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.819+0000 7fdcf7016640 10 monclient(hunting): picked mon.noname-b con 0x7fdcf00b1730 addr v1:172.21.15.169:6789/0 2024-03-25T18:00:52.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.819+0000 7fdcf7016640 10 monclient(hunting): picked mon.noname-a con 0x7fdcf00b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T18:00:52.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.820+0000 7fdcf7016640 10 monclient(hunting): picked mon.noname-c con 0x7fdcf014be90 addr v1:172.21.15.18:6790/0 2024-03-25T18:00:52.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.820+0000 7fdcf7016640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf013bd10 con 0x7fdcf00b4d60 2024-03-25T18:00:52.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.820+0000 7fdcf7016640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf0141c30 con 0x7fdcf00b1730 2024-03-25T18:00:52.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.820+0000 7fdcf7016640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf0141f10 con 0x7fdcf014be90 2024-03-25T18:00:52.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.820+0000 7fdcf7016640 10 monclient(hunting): _renew_subs 2024-03-25T18:00:52.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.820+0000 7fdcf7016640 10 monclient(hunting): authenticate will time out at 7181.398697s 2024-03-25T18:00:52.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.826+0000 7fdcf4db2640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fdcf00b1730 0x7fdcf01341f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:53796/0 (socket says 172.21.15.18:53796) 2024-03-25T18:00:52.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.826+0000 7fdcf4db2640 1 -- 172.21.15.18:0/3478398684 learned_addr learned my addr 172.21.15.18:0/3478398684 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:00:52.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.828+0000 7fdced7fa640 10 client.?.objecter ms_handle_connect 0x7fdcf00b1730 2024-03-25T18:00:52.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.828+0000 7fdced7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:52.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.832+0000 7fdced7fa640 10 client.?.objecter ms_handle_connect 0x7fdcf014be90 2024-03-25T18:00:52.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.832+0000 7fdced7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:52.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.836+0000 7fdced7fa640 1 -- 172.21.15.18:0/3478398684 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342042107 0 0) 0x7fdcf0141c30 con 0x7fdcf00b1730 2024-03-25T18:00:52.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.836+0000 7fdced7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:00:52.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.836+0000 7fdced7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:52.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.836+0000 7fdced7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:52.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.836+0000 7fdced7fa640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:00:52.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.836+0000 7fdced7fa640 1 -- 172.21.15.18:0/3478398684 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd4004eb0 con 0x7fdcf00b1730 2024-03-25T18:00:52.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.837+0000 7fdced7fa640 10 client.?.objecter ms_handle_connect 0x7fdcf00b4d60 2024-03-25T18:00:52.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.837+0000 7fdced7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:00:52.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.846+0000 7fdced7fa640 1 -- 172.21.15.18:0/3478398684 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606928028 0 0) 0x7fdcf0141f10 con 0x7fdcf014be90 2024-03-25T18:00:52.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.846+0000 7fdced7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:00:52.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.846+0000 7fdced7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:00:52.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.846+0000 7fdced7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:00:52.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.846+0000 7fdced7fa640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:00:52.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.846+0000 7fdced7fa640 1 -- 172.21.15.18:0/3478398684 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcf0141c30 con 0x7fdcf014be90 2024-03-25T18:00:52.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.847+0000 7fdced7fa640 1 -- 172.21.15.18:0/3478398684 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2754573220 0 0) 0x7fdcd4004eb0 con 0x7fdcf00b1730 2024-03-25T18:00:52.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.847+0000 7fdced7fa640 1 -- 172.21.15.18:0/3478398684 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcf0141f10 con 0x7fdcf00b1730 2024-03-25T18:00:52.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.847+0000 7fdced7fa640 1 -- 172.21.15.18:0/3478398684 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fdce40019d0 con 0x7fdcf00b1730 2024-03-25T18:00:52.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.847+0000 7fdced7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:52.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.847+0000 7fdced7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:00:52.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.847+0000 7fdced7fa640 10 monclient(hunting): dump: 2024-03-25T18:00:52.848 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:52.848 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:52.848 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:52.848 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:52.848 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:52.849 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:52.849 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:52.849 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:52.849 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:52.849 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:52.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.847+0000 7fdced7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:00:52.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdced7fa640 1 -- 172.21.15.18:0/3478398684 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1832085011 0 0) 0x7fdcf0141f10 con 0x7fdcf00b1730 2024-03-25T18:00:52.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdced7fa640 1 -- 172.21.15.18:0/3478398684 >> v1:172.21.15.18:6790/0 conn(0x7fdcf014be90 legacy=0x7fdcf01e35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:52.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdced7fa640 1 -- 172.21.15.18:0/3478398684 >> v1:172.21.15.18:6789/0 conn(0x7fdcf00b4d60 legacy=0x7fdcf0142330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:52.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdced7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:00:52.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdced7fa640 1 monclient: found mon.b 2024-03-25T18:00:52.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdced7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:00:52.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdced7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:52.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdced7fa640 1 -- 172.21.15.18:0/3478398684 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf0140810 con 0x7fdcf00b1730 2024-03-25T18:00:52.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdced7fa640 10 monclient: _finish_auth 0 2024-03-25T18:00:52.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdced7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:00:52.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdced7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:00:52.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdcf7016640 5 monclient: authenticate success, global_id 4417 2024-03-25T18:00:52.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdcf7016640 10 monclient: _renew_subs 2024-03-25T18:00:52.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdcf7016640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:52.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdcf7016640 1 -- 172.21.15.18:0/3478398684 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf0140d30 con 0x7fdcf00b1730 2024-03-25T18:00:52.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdcf7016640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:00:52.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdcf7016640 10 monclient: _renew_subs 2024-03-25T18:00:52.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdcf7016640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:52.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdcf7016640 1 -- 172.21.15.18:0/3478398684 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf0140560 con 0x7fdcf00b1730 2024-03-25T18:00:52.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.860+0000 7fdcf7016640 1 librados: init done 2024-03-25T18:00:52.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.861+0000 7fdcf7016640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:52.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.861+0000 7fdcf7016640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:52.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.861+0000 7fdcf7016640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:52.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:52.861+0000 7fdcf7016640 1 -- 172.21.15.18:0/3478398684 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcf00627b0 con 0x7fdcf00b1730 2024-03-25T18:00:53.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.138+0000 7fdced7fa640 1 -- 172.21.15.18:0/3478398684 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdce40035f0 con 0x7fdcf00b1730 2024-03-25T18:00:53.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.138+0000 7fdced7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:00:53.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.138+0000 7fdced7fa640 1 -- 172.21.15.18:0/3478398684 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fdce4003a00 con 0x7fdcf00b1730 2024-03-25T18:00:53.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.138+0000 7fdced7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:00:53.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.139+0000 7fdced7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:00:53.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.139+0000 7fdced7fa640 10 monclient: dump: 2024-03-25T18:00:53.140 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:00:53.141 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:00:53.141 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:53.141 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:00:53.141 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:00:53.141 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:00:53.141 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:00:53.141 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:00:53.141 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:00:53.141 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:00:53.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.138+0000 7fdceeffd640 4 set_mon_vals no callback set 2024-03-25T18:00:53.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.139+0000 7fdced7fa640 1 -- 172.21.15.18:0/3478398684 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fdce4003c20 con 0x7fdcf00b1730 2024-03-25T18:00:53.142 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.140+0000 7fdceffff640 10 monclient: get_auth_request con 0x7fdcd406dd10 auth_method 0 2024-03-25T18:00:53.142 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.140+0000 7fdced7fa640 10 client.4417.objecter ms_handle_connect 0x7fdcd406dd10 2024-03-25T18:00:53.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.145+0000 7fdced7fa640 1 -- 172.21.15.18:0/3478398684 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fdce40547d0 con 0x7fdcf00b1730 2024-03-25T18:00:53.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.146+0000 7fdced7fa640 10 client.4417.objecter ms_dispatch 0x7fdcf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:00:53.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.146+0000 7fdced7fa640 3 client.4417.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:00:53.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.146+0000 7fdced7fa640 3 client.4417.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:00:53.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.146+0000 7fdced7fa640 20 client.4417.objecter dump_active .. 0 homeless 2024-03-25T18:00:53.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.146+0000 7fdced7fa640 1 -- 172.21.15.18:0/3478398684 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdce40502e0 con 0x7fdcf00b1730 2024-03-25T18:00:53.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.146+0000 7fdced7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:00:53.147 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.146+0000 7fdced7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:00:53.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.373+0000 7fdcf7016640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:53.375 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.373+0000 7fdcf7016640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:53.375 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.373+0000 7fdcf7016640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:00:53.375 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.373+0000 7fdcf7016640 1 -- 172.21.15.18:0/3478398684 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcf014bbc0 con 0x7fdcf00b1730 2024-03-25T18:00:53.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.507+0000 7fdced7fa640 1 -- 172.21.15.18:0/3478398684 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fdce4052210 con 0x7fdcf00b1730 2024-03-25T18:00:53.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.507+0000 7fdced7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:00:53.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.507+0000 7fdced7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:00:53.508 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:00:53.508 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:00:53.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.509+0000 7fdcf7016640 10 librados: watch_flush enter 2024-03-25T18:00:53.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.509+0000 7fdcf7016640 10 librados: watch_flush exit 2024-03-25T18:00:53.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.509+0000 7fdcf7016640 20 client.4417.objecter shutdown clearing up homeless session... 2024-03-25T18:00:53.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.509+0000 7fdcf7016640 10 client.4417.objecter successfully canceled tick 2024-03-25T18:00:53.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.509+0000 7fdcf7016640 1 -- 172.21.15.18:0/3478398684 >> v1:172.21.15.18:6800/3721244451 conn(0x7fdcd406dd10 legacy=0x7fdcd40701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:53.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.509+0000 7fdcf7016640 10 monclient: shutdown 2024-03-25T18:00:53.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.509+0000 7fdcf7016640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:00:53.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.509+0000 7fdcf7016640 1 -- 172.21.15.18:0/3478398684 >> v1:172.21.15.169:6789/0 conn(0x7fdcf00b1730 legacy=0x7fdcf01341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:00:53.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.509+0000 7fdcf7016640 1 -- 172.21.15.18:0/3478398684 shutdown_connections 2024-03-25T18:00:53.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.509+0000 7fdcf7016640 1 -- 172.21.15.18:0/3478398684 >> 172.21.15.18:0/3478398684 conn(0x7fdcf00617a0 msgr2=0x7fdcf00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:00:53.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.509+0000 7fdcf7016640 1 -- 172.21.15.18:0/3478398684 shutdown_connections 2024-03-25T18:00:53.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.509+0000 7fdcf7016640 1 -- 172.21.15.18:0/3478398684 wait complete. 2024-03-25T18:00:53.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:00:53.509+0000 7fdcf7016640 1 librados: shutdown 2024-03-25T18:00:53.522 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:01:00.329 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:01:00.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.442+0000 7fa49e915640 10 monclient: get_monmap_and_config 2024-03-25T18:01:00.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.442+0000 7fa49e915640 10 monclient: build_initial_monmap 2024-03-25T18:01:00.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.442+0000 7fa49e915640 10 monclient: monmap: 2024-03-25T18:01:00.444 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:01:00.444 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:00.444 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:01:00.444144+0000 2024-03-25T18:01:00.444 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:01:00.444144+0000 2024-03-25T18:01:00.444 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:01:00.444 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:00.444 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:01:00.444 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:01:00.445 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:01:00.445 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:00.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.443+0000 7fa49e915640 1 Processor -- start 2024-03-25T18:01:00.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.443+0000 7fa49e915640 1 -- start start 2024-03-25T18:01:00.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.443+0000 7fa49e915640 10 monclient: init 2024-03-25T18:01:00.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.444+0000 7fa49e915640 10 monclient: _reopen_session rank -1 2024-03-25T18:01:00.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.444+0000 7fa49e915640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:01:00.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.444+0000 7fa49e915640 10 monclient(hunting): picked mon.noname-c con 0x7fa4980a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:01:00.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.444+0000 7fa49e915640 10 monclient(hunting): picked mon.noname-a con 0x7fa49813f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:01:00.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.444+0000 7fa49e915640 10 monclient(hunting): picked mon.noname-b con 0x7fa4981430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:01:00.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.444+0000 7fa49e915640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa498142520 con 0x7fa49813f710 2024-03-25T18:01:00.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.444+0000 7fa49e915640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4980b28a0 con 0x7fa4980a98f0 2024-03-25T18:01:00.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.444+0000 7fa49e915640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4980b2a30 con 0x7fa4981430c0 2024-03-25T18:01:00.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.444+0000 7fa49e915640 10 monclient(hunting): _renew_subs 2024-03-25T18:01:00.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.444+0000 7fa49e915640 10 monclient(hunting): authenticate will time out at 7189.023171s 2024-03-25T18:01:00.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.450+0000 7fa497fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fa4980a98f0 0x7fa4980a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:42168/0 (socket says 172.21.15.18:42168) 2024-03-25T18:01:00.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.450+0000 7fa497fff640 1 -- 172.21.15.18:0/4151444706 learned_addr learned my addr 172.21.15.18:0/4151444706 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:01:00.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.462+0000 7fa496ffd640 1 -- 172.21.15.18:0/4151444706 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454501534 0 0) 0x7fa4980b28a0 con 0x7fa4980a98f0 2024-03-25T18:01:00.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.462+0000 7fa496ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:01:00.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.462+0000 7fa496ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:00.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.462+0000 7fa496ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:00.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.462+0000 7fa496ffd640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:01:00.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.462+0000 7fa496ffd640 1 -- 172.21.15.18:0/4151444706 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa480003520 con 0x7fa4980a98f0 2024-03-25T18:01:00.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.463+0000 7fa496ffd640 1 -- 172.21.15.18:0/4151444706 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755121907 0 0) 0x7fa498142520 con 0x7fa49813f710 2024-03-25T18:01:00.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.463+0000 7fa496ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:01:00.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.463+0000 7fa496ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:00.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.463+0000 7fa496ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:00.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.463+0000 7fa496ffd640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:01:00.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.463+0000 7fa496ffd640 1 -- 172.21.15.18:0/4151444706 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4980b28a0 con 0x7fa49813f710 2024-03-25T18:01:00.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa496ffd640 1 -- 172.21.15.18:0/4151444706 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2034150602 0 0) 0x7fa480003520 con 0x7fa4980a98f0 2024-03-25T18:01:00.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa496ffd640 1 -- 172.21.15.18:0/4151444706 >> v1:172.21.15.18:6790/0 conn(0x7fa4981430c0 legacy=0x7fa49814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:00.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa496ffd640 1 -- 172.21.15.18:0/4151444706 >> v1:172.21.15.18:6789/0 conn(0x7fa49813f710 legacy=0x7fa4981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:00.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa496ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:01:00.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa496ffd640 1 monclient: found mon.noname-c 2024-03-25T18:01:00.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa496ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:01:00.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa496ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:01:00.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa496ffd640 1 -- 172.21.15.18:0/4151444706 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4980b44c0 con 0x7fa4980a98f0 2024-03-25T18:01:00.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa496ffd640 10 monclient: _finish_auth 0 2024-03-25T18:01:00.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa496ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:01:00.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa496ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:01:00.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa496ffd640 1 -- 172.21.15.18:0/4151444706 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa488001690 con 0x7fa4980a98f0 2024-03-25T18:01:00.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa49e915640 5 monclient: authenticate success, global_id 4420 2024-03-25T18:01:00.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa49e915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:01:00.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa496ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:00.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa496ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:01:00.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa496ffd640 10 monclient: dump: 2024-03-25T18:01:00.474 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:00.474 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:00.475 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:00.475 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:00.475 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:00.475 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:00.475 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:00.475 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:00.475 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:00.475 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:00.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.472+0000 7fa49e915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:01:00.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.479+0000 7fa496ffd640 1 -- 172.21.15.18:0/4151444706 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4880017f0 con 0x7fa4980a98f0 2024-03-25T18:01:00.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.480+0000 7fa496ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:01:00.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.480+0000 7fa496ffd640 1 -- 172.21.15.18:0/4151444706 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa488001c00 con 0x7fa4980a98f0 2024-03-25T18:01:00.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.480+0000 7fa49e915640 10 monclient: get_monmap_and_config success 2024-03-25T18:01:00.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.480+0000 7fa49e915640 4 set_mon_vals no callback set 2024-03-25T18:01:00.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.480+0000 7fa496ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:00.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.480+0000 7fa496ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:01:00.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.480+0000 7fa496ffd640 10 monclient: dump: 2024-03-25T18:01:00.481 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:00.481 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:00.481 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:00.481 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:00.481 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:00.481 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:00.482 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:00.482 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:00.482 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:00.482 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:00.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.480+0000 7fa49e915640 10 monclient: shutdown 2024-03-25T18:01:00.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.480+0000 7fa49e915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:01:00.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.480+0000 7fa49e915640 1 -- 172.21.15.18:0/4151444706 >> v1:172.21.15.169:6789/0 conn(0x7fa4980a98f0 legacy=0x7fa4980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:00.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.481+0000 7fa49e915640 1 -- 172.21.15.18:0/4151444706 shutdown_connections 2024-03-25T18:01:00.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.481+0000 7fa49e915640 1 -- 172.21.15.18:0/4151444706 >> 172.21.15.18:0/4151444706 conn(0x7fa4980b7ea0 msgr2=0x7fa4980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:01:00.483 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.481+0000 7fa49e915640 1 -- 172.21.15.18:0/4151444706 shutdown_connections 2024-03-25T18:01:00.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.481+0000 7fa49e915640 1 -- 172.21.15.18:0/4151444706 wait complete. 2024-03-25T18:01:00.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.482+0000 7fa49e915640 10 monclient: build_initial_monmap 2024-03-25T18:01:00.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.482+0000 7fa49e915640 10 monclient: monmap: 2024-03-25T18:01:00.484 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:01:00.484 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:01:00.484 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:01:00.484 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:01:00.484 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:01:00.484 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:01:00.484 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:01:00.484 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:01:00.484 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:01:00.484 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:00.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.482+0000 7fa49e915640 1 librados: starting msgr at 2024-03-25T18:01:00.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.482+0000 7fa49e915640 1 librados: starting objecter 2024-03-25T18:01:00.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.482+0000 7fa49e915640 1 Processor -- start 2024-03-25T18:01:00.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.482+0000 7fa49e915640 1 -- start start 2024-03-25T18:01:00.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.482+0000 7fa49e915640 1 librados: setting wanted keys 2024-03-25T18:01:00.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.482+0000 7fa49e915640 1 librados: calling monclient init 2024-03-25T18:01:00.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.482+0000 7fa49e915640 10 monclient: init 2024-03-25T18:01:00.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.483+0000 7fa49e915640 10 monclient: _reopen_session rank -1 2024-03-25T18:01:00.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.483+0000 7fa49e915640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:01:00.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.483+0000 7fa49e915640 10 monclient(hunting): picked mon.noname-c con 0x7fa4980a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:01:00.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.483+0000 7fa49e915640 10 monclient(hunting): picked mon.noname-a con 0x7fa49813f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:01:00.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.483+0000 7fa49e915640 10 monclient(hunting): picked mon.noname-b con 0x7fa4981430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:01:00.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.483+0000 7fa49e915640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4981d7380 con 0x7fa49813f710 2024-03-25T18:01:00.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.483+0000 7fa49e915640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa498139840 con 0x7fa4981430c0 2024-03-25T18:01:00.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.483+0000 7fa49e915640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4981d8200 con 0x7fa4980a98f0 2024-03-25T18:01:00.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.483+0000 7fa49e915640 10 monclient(hunting): _renew_subs 2024-03-25T18:01:00.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.483+0000 7fa49e915640 10 monclient(hunting): authenticate will time out at 7189.062404s 2024-03-25T18:01:00.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.490+0000 7fa4977fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fa49813f710 0x7fa4981da880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:44156/0 (socket says 172.21.15.18:44156) 2024-03-25T18:01:00.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.490+0000 7fa4977fe640 1 -- 172.21.15.18:0/3451333528 learned_addr learned my addr 172.21.15.18:0/3451333528 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:01:00.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.493+0000 7fa494ff9640 10 client.?.objecter ms_handle_connect 0x7fa49813f710 2024-03-25T18:01:00.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.493+0000 7fa494ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:00.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.503+0000 7fa494ff9640 10 client.?.objecter ms_handle_connect 0x7fa4981430c0 2024-03-25T18:01:00.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.503+0000 7fa494ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:00.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.506+0000 7fa494ff9640 10 client.?.objecter ms_handle_connect 0x7fa4980a98f0 2024-03-25T18:01:00.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.506+0000 7fa494ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:00.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.513+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848200680 0 0) 0x7fa4981d7380 con 0x7fa49813f710 2024-03-25T18:01:00.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.513+0000 7fa494ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:01:00.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.513+0000 7fa494ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:00.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.513+0000 7fa494ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:00.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.513+0000 7fa494ff9640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:01:00.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.513+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa484004eb0 con 0x7fa49813f710 2024-03-25T18:01:00.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.516+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054393691 0 0) 0x7fa498139840 con 0x7fa4981430c0 2024-03-25T18:01:00.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.516+0000 7fa494ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:01:00.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.516+0000 7fa494ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:00.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.516+0000 7fa494ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:00.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.516+0000 7fa494ff9640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:01:00.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.516+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4981d7380 con 0x7fa4981430c0 2024-03-25T18:01:00.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.525+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3684667252 0 0) 0x7fa484004eb0 con 0x7fa49813f710 2024-03-25T18:01:00.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.525+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa498139840 con 0x7fa49813f710 2024-03-25T18:01:00.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.525+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa48c001a10 con 0x7fa49813f710 2024-03-25T18:01:00.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.525+0000 7fa494ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:00.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.525+0000 7fa494ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:01:00.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.525+0000 7fa494ff9640 10 monclient(hunting): dump: 2024-03-25T18:01:00.526 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:00.526 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:00.526 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:00.526 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:00.526 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:00.526 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:00.526 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:00.526 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:00.526 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:00.527 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:00.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.525+0000 7fa494ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:01:00.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.526+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850616527 0 0) 0x7fa4981d8200 con 0x7fa4980a98f0 2024-03-25T18:01:00.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.526+0000 7fa494ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:01:00.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.526+0000 7fa494ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:00.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.526+0000 7fa494ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:00.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.526+0000 7fa494ff9640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:01:00.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.526+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa484004eb0 con 0x7fa4980a98f0 2024-03-25T18:01:00.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.540+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2589980103 0 0) 0x7fa4981d7380 con 0x7fa4981430c0 2024-03-25T18:01:00.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.540+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4981d8200 con 0x7fa4981430c0 2024-03-25T18:01:00.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.540+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa490001b10 con 0x7fa4981430c0 2024-03-25T18:01:00.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.540+0000 7fa494ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:00.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.540+0000 7fa494ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:01:00.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.540+0000 7fa494ff9640 10 monclient(hunting): dump: 2024-03-25T18:01:00.542 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:00.542 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:00.542 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:00.542 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:00.542 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:00.542 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:00.542 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:00.542 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:00.542 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:00.542 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:00.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.542+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3046440932 0 0) 0x7fa484004eb0 con 0x7fa4980a98f0 2024-03-25T18:01:00.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.542+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4981d7380 con 0x7fa4980a98f0 2024-03-25T18:01:00.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.542+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa488001b00 con 0x7fa4980a98f0 2024-03-25T18:01:00.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.542+0000 7fa494ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:00.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.542+0000 7fa494ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:01:00.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.542+0000 7fa494ff9640 10 monclient(hunting): dump: 2024-03-25T18:01:00.543 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:00.544 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:00.544 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:00.544 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:00.544 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:00.544 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:00.544 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:00.544 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:00.544 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:00.544 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:00.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3711139228 0 0) 0x7fa498139840 con 0x7fa49813f710 2024-03-25T18:01:00.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 >> v1:172.21.15.18:6790/0 conn(0x7fa4980a98f0 legacy=0x7fa4981d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:00.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 >> v1:172.21.15.169:6789/0 conn(0x7fa4981430c0 legacy=0x7fa4981daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:00.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa494ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:01:00.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa494ff9640 1 monclient: found mon.a 2024-03-25T18:01:00.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa494ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:01:00.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa494ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:01:00.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4981d75d0 con 0x7fa49813f710 2024-03-25T18:01:00.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa494ff9640 10 monclient: _finish_auth 0 2024-03-25T18:01:00.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa494ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:01:00.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa494ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:01:00.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa49e915640 5 monclient: authenticate success, global_id 4434 2024-03-25T18:01:00.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa49e915640 10 monclient: _renew_subs 2024-03-25T18:01:00.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa49e915640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:01:00.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa49e915640 1 -- 172.21.15.18:0/3451333528 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4981db6a0 con 0x7fa49813f710 2024-03-25T18:01:00.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa49e915640 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:01:00.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa49e915640 10 monclient: _renew_subs 2024-03-25T18:01:00.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa49e915640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:01:00.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa49e915640 1 -- 172.21.15.18:0/3451333528 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa498137910 con 0x7fa49813f710 2024-03-25T18:01:00.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.551+0000 7fa49e915640 1 librados: init done 2024-03-25T18:01:00.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.552+0000 7fa49e915640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:00.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.552+0000 7fa49e915640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:00.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.552+0000 7fa49e915640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:01:00.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.552+0000 7fa49e915640 1 -- 172.21.15.18:0/3451333528 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4980627b0 con 0x7fa49813f710 2024-03-25T18:01:00.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.865+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa48c001c50 con 0x7fa49813f710 2024-03-25T18:01:00.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.865+0000 7fa494ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:01:00.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.865+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa48c003a60 con 0x7fa49813f710 2024-03-25T18:01:00.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.865+0000 7fa494ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:00.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.865+0000 7fa494ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:01:00.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.865+0000 7fa494ff9640 10 monclient: dump: 2024-03-25T18:01:00.867 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:00.867 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:00.867 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:00.867 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:00.867 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:00.867 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:00.867 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:00.867 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:00.867 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:00.867 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:00.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.865+0000 7fa4967fc640 4 set_mon_vals no callback set 2024-03-25T18:01:00.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.866+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fa48c003c80 con 0x7fa49813f710 2024-03-25T18:01:00.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.866+0000 7fa497fff640 10 monclient: get_auth_request con 0x7fa48406e180 auth_method 0 2024-03-25T18:01:00.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.867+0000 7fa494ff9640 10 client.4434.objecter ms_handle_connect 0x7fa48406e180 2024-03-25T18:01:00.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.874+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fa48c054850 con 0x7fa49813f710 2024-03-25T18:01:00.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.874+0000 7fa494ff9640 10 client.4434.objecter ms_dispatch 0x7fa498000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:01:00.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.874+0000 7fa494ff9640 3 client.4434.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:01:00.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.874+0000 7fa494ff9640 3 client.4434.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:01:00.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.874+0000 7fa494ff9640 20 client.4434.objecter dump_active .. 0 homeless 2024-03-25T18:01:00.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.874+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa48c050360 con 0x7fa49813f710 2024-03-25T18:01:00.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.874+0000 7fa494ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:00.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:00.874+0000 7fa494ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:01:01.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.122+0000 7fa49e915640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:01.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.122+0000 7fa49e915640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:01.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.122+0000 7fa49e915640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:01:01.124 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.122+0000 7fa49e915640 1 -- 172.21.15.18:0/3451333528 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa498069f40 con 0x7fa49813f710 2024-03-25T18:01:01.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.155+0000 7fa494ff9640 1 -- 172.21.15.18:0/3451333528 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fa48c052290 con 0x7fa49813f710 2024-03-25T18:01:01.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.155+0000 7fa494ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:01.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.155+0000 7fa494ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:01:01.157 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:01:01.157 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:01:01.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.157+0000 7fa49e915640 10 librados: watch_flush enter 2024-03-25T18:01:01.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.157+0000 7fa49e915640 10 librados: watch_flush exit 2024-03-25T18:01:01.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.157+0000 7fa49e915640 20 client.4434.objecter shutdown clearing up homeless session... 2024-03-25T18:01:01.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.157+0000 7fa49e915640 10 client.4434.objecter successfully canceled tick 2024-03-25T18:01:01.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.157+0000 7fa49e915640 1 -- 172.21.15.18:0/3451333528 >> v1:172.21.15.18:6800/3721244451 conn(0x7fa48406e180 legacy=0x7fa484070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:01.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.157+0000 7fa49e915640 10 monclient: shutdown 2024-03-25T18:01:01.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.157+0000 7fa49e915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:01:01.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.157+0000 7fa49e915640 1 -- 172.21.15.18:0/3451333528 >> v1:172.21.15.18:6789/0 conn(0x7fa49813f710 legacy=0x7fa4981da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:01.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.157+0000 7fa49e915640 1 -- 172.21.15.18:0/3451333528 shutdown_connections 2024-03-25T18:01:01.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.157+0000 7fa49e915640 1 -- 172.21.15.18:0/3451333528 >> 172.21.15.18:0/3451333528 conn(0x7fa4980b7ea0 msgr2=0x7fa498074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:01:01.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.158+0000 7fa49e915640 1 -- 172.21.15.18:0/3451333528 shutdown_connections 2024-03-25T18:01:01.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.158+0000 7fa49e915640 1 -- 172.21.15.18:0/3451333528 wait complete. 2024-03-25T18:01:01.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:01.158+0000 7fa49e915640 1 librados: shutdown 2024-03-25T18:01:01.170 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:01:02.862 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:00.654 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:02.878 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:00.671 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:02.914 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:00.707 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:02.927 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:00.677 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:02.938 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:00.731 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:02.940 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:00.689 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:02.949 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:00.742 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:02.975 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:00.768 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:02.980 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:00.729 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:07.977 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:01:08.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.097+0000 7fd855a4a640 10 monclient: get_monmap_and_config 2024-03-25T18:01:08.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.097+0000 7fd855a4a640 10 monclient: build_initial_monmap 2024-03-25T18:01:08.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.097+0000 7fd855a4a640 10 monclient: monmap: 2024-03-25T18:01:08.097 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:01:08.097 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:08.097 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:01:08.097500+0000 2024-03-25T18:01:08.097 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:01:08.097500+0000 2024-03-25T18:01:08.097 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:01:08.097 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:08.098 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:01:08.098 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:01:08.098 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:01:08.098 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:08.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.097+0000 7fd855a4a640 1 Processor -- start 2024-03-25T18:01:08.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.098+0000 7fd855a4a640 1 -- start start 2024-03-25T18:01:08.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.098+0000 7fd855a4a640 10 monclient: init 2024-03-25T18:01:08.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.098+0000 7fd855a4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:01:08.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.098+0000 7fd855a4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:01:08.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.098+0000 7fd855a4a640 10 monclient(hunting): picked mon.noname-c con 0x7fd8500a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:01:08.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.098+0000 7fd855a4a640 10 monclient(hunting): picked mon.noname-a con 0x7fd85013f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:01:08.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.098+0000 7fd855a4a640 10 monclient(hunting): picked mon.noname-b con 0x7fd8501430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:01:08.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.098+0000 7fd855a4a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd850142520 con 0x7fd85013f710 2024-03-25T18:01:08.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.098+0000 7fd855a4a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8500b28a0 con 0x7fd8500a98f0 2024-03-25T18:01:08.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.098+0000 7fd855a4a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8500b2a30 con 0x7fd8501430c0 2024-03-25T18:01:08.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.098+0000 7fd855a4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:01:08.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.098+0000 7fd855a4a640 10 monclient(hunting): authenticate will time out at 7196.676447s 2024-03-25T18:01:08.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.100+0000 7fd84effd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fd8500a98f0 0x7fd8500a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:58758/0 (socket says 172.21.15.18:58758) 2024-03-25T18:01:08.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.100+0000 7fd84effd640 1 -- 172.21.15.18:0/2119768927 learned_addr learned my addr 172.21.15.18:0/2119768927 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:01:08.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.201+0000 7fd84dffb640 1 -- 172.21.15.18:0/2119768927 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417252541 0 0) 0x7fd850142520 con 0x7fd85013f710 2024-03-25T18:01:08.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.201+0000 7fd84dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:01:08.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.201+0000 7fd84dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:08.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.201+0000 7fd84dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:08.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.201+0000 7fd84dffb640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:01:08.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.201+0000 7fd84dffb640 1 -- 172.21.15.18:0/2119768927 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd838003520 con 0x7fd85013f710 2024-03-25T18:01:08.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.219+0000 7fd84dffb640 1 -- 172.21.15.18:0/2119768927 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3761066493 0 0) 0x7fd838003520 con 0x7fd85013f710 2024-03-25T18:01:08.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.219+0000 7fd84dffb640 1 -- 172.21.15.18:0/2119768927 >> v1:172.21.15.18:6790/0 conn(0x7fd8501430c0 legacy=0x7fd85014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:08.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.219+0000 7fd84dffb640 1 -- 172.21.15.18:0/2119768927 >> v1:172.21.15.169:6789/0 conn(0x7fd8500a98f0 legacy=0x7fd8500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:08.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.219+0000 7fd84dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:01:08.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.219+0000 7fd84dffb640 1 monclient: found mon.noname-a 2024-03-25T18:01:08.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.219+0000 7fd84dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:01:08.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.219+0000 7fd84dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:01:08.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.219+0000 7fd84dffb640 1 -- 172.21.15.18:0/2119768927 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8500b44c0 con 0x7fd85013f710 2024-03-25T18:01:08.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.219+0000 7fd84dffb640 10 monclient: _finish_auth 0 2024-03-25T18:01:08.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.219+0000 7fd84dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:01:08.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.219+0000 7fd84dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:01:08.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.219+0000 7fd84dffb640 1 -- 172.21.15.18:0/2119768927 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd844001690 con 0x7fd85013f710 2024-03-25T18:01:08.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.219+0000 7fd84dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:08.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.219+0000 7fd84dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:01:08.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.219+0000 7fd84dffb640 10 monclient: dump: 2024-03-25T18:01:08.220 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:08.220 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:08.220 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:08.220 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:08.221 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:08.221 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:08.221 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:08.221 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:08.221 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:08.221 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:08.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.219+0000 7fd855a4a640 5 monclient: authenticate success, global_id 4437 2024-03-25T18:01:08.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.219+0000 7fd855a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:01:08.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.232+0000 7fd84dffb640 1 -- 172.21.15.18:0/2119768927 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8440017f0 con 0x7fd85013f710 2024-03-25T18:01:08.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.232+0000 7fd84dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:01:08.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.232+0000 7fd84dffb640 1 -- 172.21.15.18:0/2119768927 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd844001c00 con 0x7fd85013f710 2024-03-25T18:01:08.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.232+0000 7fd84dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:08.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.232+0000 7fd84dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:01:08.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.232+0000 7fd84dffb640 10 monclient: dump: 2024-03-25T18:01:08.232 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:08.232 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:08.232 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:08.233 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:08.233 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:08.233 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:08.233 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:08.233 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:08.233 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:08.233 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:08.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.232+0000 7fd855a4a640 10 monclient: get_monmap_and_config success 2024-03-25T18:01:08.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.232+0000 7fd855a4a640 4 set_mon_vals no callback set 2024-03-25T18:01:08.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.232+0000 7fd855a4a640 10 monclient: shutdown 2024-03-25T18:01:08.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.232+0000 7fd855a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:01:08.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.232+0000 7fd855a4a640 1 -- 172.21.15.18:0/2119768927 >> v1:172.21.15.18:6789/0 conn(0x7fd85013f710 legacy=0x7fd8501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:08.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.232+0000 7fd855a4a640 1 -- 172.21.15.18:0/2119768927 shutdown_connections 2024-03-25T18:01:08.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.232+0000 7fd855a4a640 1 -- 172.21.15.18:0/2119768927 >> 172.21.15.18:0/2119768927 conn(0x7fd8500b7ea0 msgr2=0x7fd8500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:01:08.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.233+0000 7fd855a4a640 1 -- 172.21.15.18:0/2119768927 shutdown_connections 2024-03-25T18:01:08.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.233+0000 7fd855a4a640 1 -- 172.21.15.18:0/2119768927 wait complete. 2024-03-25T18:01:08.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.234+0000 7fd855a4a640 10 monclient: build_initial_monmap 2024-03-25T18:01:08.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.234+0000 7fd855a4a640 10 monclient: monmap: 2024-03-25T18:01:08.235 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:01:08.235 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:01:08.235 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:01:08.235 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:01:08.235 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:01:08.235 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:01:08.235 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:01:08.236 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:01:08.236 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:01:08.236 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:08.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.234+0000 7fd855a4a640 1 librados: starting msgr at 2024-03-25T18:01:08.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.234+0000 7fd855a4a640 1 librados: starting objecter 2024-03-25T18:01:08.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.234+0000 7fd855a4a640 1 Processor -- start 2024-03-25T18:01:08.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.234+0000 7fd855a4a640 1 -- start start 2024-03-25T18:01:08.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.234+0000 7fd855a4a640 1 librados: setting wanted keys 2024-03-25T18:01:08.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.234+0000 7fd855a4a640 1 librados: calling monclient init 2024-03-25T18:01:08.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.234+0000 7fd855a4a640 10 monclient: init 2024-03-25T18:01:08.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.235+0000 7fd855a4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:01:08.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.235+0000 7fd855a4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:01:08.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.235+0000 7fd855a4a640 10 monclient(hunting): picked mon.noname-a con 0x7fd8500a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:01:08.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.235+0000 7fd855a4a640 10 monclient(hunting): picked mon.noname-c con 0x7fd85013f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:01:08.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.235+0000 7fd855a4a640 10 monclient(hunting): picked mon.noname-b con 0x7fd8501430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:01:08.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.235+0000 7fd855a4a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8500abd70 con 0x7fd8500a98f0 2024-03-25T18:01:08.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.235+0000 7fd855a4a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8501d7710 con 0x7fd8501430c0 2024-03-25T18:01:08.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.235+0000 7fd855a4a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8501d8340 con 0x7fd85013f710 2024-03-25T18:01:08.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.235+0000 7fd855a4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:01:08.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.235+0000 7fd855a4a640 10 monclient(hunting): authenticate will time out at 7196.813463s 2024-03-25T18:01:08.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.237+0000 7fd84f7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fd8501430c0 0x7fd8501d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:58792/0 (socket says 172.21.15.18:58792) 2024-03-25T18:01:08.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.237+0000 7fd84f7fe640 1 -- 172.21.15.18:0/3764960451 learned_addr learned my addr 172.21.15.18:0/3764960451 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:01:08.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.239+0000 7fd837fff640 10 client.?.objecter ms_handle_connect 0x7fd8501430c0 2024-03-25T18:01:08.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.239+0000 7fd837fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:08.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.243+0000 7fd837fff640 1 -- 172.21.15.18:0/3764960451 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744330864 0 0) 0x7fd8501d7710 con 0x7fd8501430c0 2024-03-25T18:01:08.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.243+0000 7fd837fff640 10 monclient(hunting): _negotiate 2024-03-25T18:01:08.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.243+0000 7fd837fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:08.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.243+0000 7fd837fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:08.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.243+0000 7fd837fff640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:01:08.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.243+0000 7fd837fff640 1 -- 172.21.15.18:0/3764960451 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd83c004eb0 con 0x7fd8501430c0 2024-03-25T18:01:08.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.244+0000 7fd837fff640 10 client.?.objecter ms_handle_connect 0x7fd85013f710 2024-03-25T18:01:08.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.244+0000 7fd837fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:08.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.252+0000 7fd837fff640 1 -- 172.21.15.18:0/3764960451 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1664816567 0 0) 0x7fd83c004eb0 con 0x7fd8501430c0 2024-03-25T18:01:08.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.252+0000 7fd837fff640 1 -- 172.21.15.18:0/3764960451 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8501d7710 con 0x7fd8501430c0 2024-03-25T18:01:08.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.252+0000 7fd837fff640 1 -- 172.21.15.18:0/3764960451 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd848001c30 con 0x7fd8501430c0 2024-03-25T18:01:08.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.252+0000 7fd837fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:08.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.252+0000 7fd837fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:01:08.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.252+0000 7fd837fff640 10 monclient(hunting): dump: 2024-03-25T18:01:08.253 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:08.253 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:08.253 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:08.253 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:08.253 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:08.254 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:08.254 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:08.254 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:08.254 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:08.254 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:08.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.252+0000 7fd837fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:01:08.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.255+0000 7fd837fff640 10 client.?.objecter ms_handle_connect 0x7fd8500a98f0 2024-03-25T18:01:08.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.255+0000 7fd837fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:08.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.255+0000 7fd837fff640 1 -- 172.21.15.18:0/3764960451 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620548658 0 0) 0x7fd8501d8340 con 0x7fd85013f710 2024-03-25T18:01:08.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.255+0000 7fd837fff640 10 monclient(hunting): _negotiate 2024-03-25T18:01:08.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.255+0000 7fd837fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:08.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.255+0000 7fd837fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:08.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.255+0000 7fd837fff640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:01:08.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.255+0000 7fd837fff640 1 -- 172.21.15.18:0/3764960451 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd83c004eb0 con 0x7fd85013f710 2024-03-25T18:01:08.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd837fff640 1 -- 172.21.15.18:0/3764960451 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3527935408 0 0) 0x7fd8501d7710 con 0x7fd8501430c0 2024-03-25T18:01:08.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd837fff640 1 -- 172.21.15.18:0/3764960451 >> v1:172.21.15.18:6790/0 conn(0x7fd85013f710 legacy=0x7fd8501d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:08.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd837fff640 1 -- 172.21.15.18:0/3764960451 >> v1:172.21.15.18:6789/0 conn(0x7fd8500a98f0 legacy=0x7fd850137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:08.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd837fff640 10 monclient: _finish_hunting 0 2024-03-25T18:01:08.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd837fff640 1 monclient: found mon.b 2024-03-25T18:01:08.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd837fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:01:08.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd837fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:08.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd837fff640 1 -- 172.21.15.18:0/3764960451 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8501d84d0 con 0x7fd8501430c0 2024-03-25T18:01:08.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd837fff640 10 monclient: _finish_auth 0 2024-03-25T18:01:08.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd837fff640 10 monclient: _check_auth_tickets 2024-03-25T18:01:08.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd837fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:01:08.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd855a4a640 5 monclient: authenticate success, global_id 4429 2024-03-25T18:01:08.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd855a4a640 10 monclient: _renew_subs 2024-03-25T18:01:08.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd855a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:08.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd855a4a640 1 -- 172.21.15.18:0/3764960451 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8501d41e0 con 0x7fd8501430c0 2024-03-25T18:01:08.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd855a4a640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:01:08.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd855a4a640 10 monclient: _renew_subs 2024-03-25T18:01:08.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd855a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:08.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd855a4a640 1 -- 172.21.15.18:0/3764960451 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8501d5150 con 0x7fd8501430c0 2024-03-25T18:01:08.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.264+0000 7fd855a4a640 1 librados: init done 2024-03-25T18:01:08.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.265+0000 7fd855a4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:08.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.265+0000 7fd855a4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:08.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.265+0000 7fd855a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:08.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.265+0000 7fd855a4a640 1 -- 172.21.15.18:0/3764960451 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8500627b0 con 0x7fd8501430c0 2024-03-25T18:01:08.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.527+0000 7fd837fff640 1 -- 172.21.15.18:0/3764960451 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd848003990 con 0x7fd8501430c0 2024-03-25T18:01:08.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.527+0000 7fd837fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:01:08.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.527+0000 7fd837fff640 1 -- 172.21.15.18:0/3764960451 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd848003da0 con 0x7fd8501430c0 2024-03-25T18:01:08.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.527+0000 7fd837fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:08.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.527+0000 7fd837fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:01:08.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.527+0000 7fd837fff640 10 monclient: dump: 2024-03-25T18:01:08.528 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:08.528 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:08.528 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:08.528 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:08.528 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:08.528 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:08.528 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:08.529 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:08.529 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:08.529 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:08.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.527+0000 7fd84d7fa640 4 set_mon_vals no callback set 2024-03-25T18:01:08.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.528+0000 7fd837fff640 1 -- 172.21.15.18:0/3764960451 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fd848003fc0 con 0x7fd8501430c0 2024-03-25T18:01:08.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.529+0000 7fd84effd640 10 monclient: get_auth_request con 0x7fd83c06dd60 auth_method 0 2024-03-25T18:01:08.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.529+0000 7fd837fff640 10 client.4429.objecter ms_handle_connect 0x7fd83c06dd60 2024-03-25T18:01:08.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.536+0000 7fd837fff640 1 -- 172.21.15.18:0/3764960451 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fd848054a90 con 0x7fd8501430c0 2024-03-25T18:01:08.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.536+0000 7fd837fff640 10 client.4429.objecter ms_dispatch 0x7fd850000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:01:08.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.536+0000 7fd837fff640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:01:08.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.536+0000 7fd837fff640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:01:08.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.536+0000 7fd837fff640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-25T18:01:08.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.536+0000 7fd837fff640 1 -- 172.21.15.18:0/3764960451 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd8480505a0 con 0x7fd8501430c0 2024-03-25T18:01:08.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.536+0000 7fd837fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:08.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.536+0000 7fd837fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:01:08.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.794+0000 7fd855a4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:08.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.794+0000 7fd855a4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:08.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.794+0000 7fd855a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:08.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.794+0000 7fd855a4a640 1 -- 172.21.15.18:0/3764960451 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd850069f40 con 0x7fd8501430c0 2024-03-25T18:01:08.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.821+0000 7fd837fff640 1 -- 172.21.15.18:0/3764960451 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fd8480524d0 con 0x7fd8501430c0 2024-03-25T18:01:08.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.821+0000 7fd837fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:08.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.821+0000 7fd837fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:01:08.821 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:01:08.822 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:01:08.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.823+0000 7fd855a4a640 10 librados: watch_flush enter 2024-03-25T18:01:08.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.823+0000 7fd855a4a640 10 librados: watch_flush exit 2024-03-25T18:01:08.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.823+0000 7fd855a4a640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-25T18:01:08.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.823+0000 7fd855a4a640 10 client.4429.objecter successfully canceled tick 2024-03-25T18:01:08.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.823+0000 7fd855a4a640 1 -- 172.21.15.18:0/3764960451 >> v1:172.21.15.18:6800/3721244451 conn(0x7fd83c06dd60 legacy=0x7fd83c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:08.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.823+0000 7fd855a4a640 10 monclient: shutdown 2024-03-25T18:01:08.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.823+0000 7fd855a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:01:08.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.823+0000 7fd855a4a640 1 -- 172.21.15.18:0/3764960451 >> v1:172.21.15.169:6789/0 conn(0x7fd8501430c0 legacy=0x7fd8501d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:08.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.823+0000 7fd855a4a640 1 -- 172.21.15.18:0/3764960451 shutdown_connections 2024-03-25T18:01:08.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.823+0000 7fd855a4a640 1 -- 172.21.15.18:0/3764960451 >> 172.21.15.18:0/3764960451 conn(0x7fd8500b7ea0 msgr2=0x7fd850074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:01:08.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.824+0000 7fd855a4a640 1 -- 172.21.15.18:0/3764960451 shutdown_connections 2024-03-25T18:01:08.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.824+0000 7fd855a4a640 1 -- 172.21.15.18:0/3764960451 wait complete. 2024-03-25T18:01:08.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:08.824+0000 7fd855a4a640 1 librados: shutdown 2024-03-25T18:01:08.837 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:01:15.644 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:01:15.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.758+0000 7fe073b63640 10 monclient: get_monmap_and_config 2024-03-25T18:01:15.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.758+0000 7fe073b63640 10 monclient: build_initial_monmap 2024-03-25T18:01:15.759 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.758+0000 7fe073b63640 10 monclient: monmap: 2024-03-25T18:01:15.759 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:01:15.759 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:15.759 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:01:15.759202+0000 2024-03-25T18:01:15.759 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:01:15.759202+0000 2024-03-25T18:01:15.759 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:01:15.759 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:15.759 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:01:15.759 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:01:15.759 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:01:15.760 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:15.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.759+0000 7fe073b63640 1 Processor -- start 2024-03-25T18:01:15.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.759+0000 7fe073b63640 1 -- start start 2024-03-25T18:01:15.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.759+0000 7fe073b63640 10 monclient: init 2024-03-25T18:01:15.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.760+0000 7fe073b63640 10 monclient: _reopen_session rank -1 2024-03-25T18:01:15.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.760+0000 7fe073b63640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:01:15.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.760+0000 7fe073b63640 10 monclient(hunting): picked mon.noname-a con 0x7fe06c0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:01:15.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.760+0000 7fe073b63640 10 monclient(hunting): picked mon.noname-c con 0x7fe06c13f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:01:15.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.760+0000 7fe073b63640 10 monclient(hunting): picked mon.noname-b con 0x7fe06c1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:01:15.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.760+0000 7fe073b63640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe06c142520 con 0x7fe06c0a98f0 2024-03-25T18:01:15.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.760+0000 7fe073b63640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe06c0b28a0 con 0x7fe06c13f710 2024-03-25T18:01:15.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.760+0000 7fe073b63640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe06c0b2a30 con 0x7fe06c1430c0 2024-03-25T18:01:15.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.760+0000 7fe073b63640 10 monclient(hunting): _renew_subs 2024-03-25T18:01:15.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.760+0000 7fe073b63640 10 monclient(hunting): authenticate will time out at 7204.338085s 2024-03-25T18:01:15.769 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.769+0000 7fe0718ff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fe06c0a98f0 0x7fe06c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:47278/0 (socket says 172.21.15.18:47278) 2024-03-25T18:01:15.769 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.769+0000 7fe0718ff640 1 -- 172.21.15.18:0/1254456252 learned_addr learned my addr 172.21.15.18:0/1254456252 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:01:15.784 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.783+0000 7fe0708fd640 1 -- 172.21.15.18:0/1254456252 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33647401 0 0) 0x7fe06c142520 con 0x7fe06c0a98f0 2024-03-25T18:01:15.784 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.783+0000 7fe0708fd640 10 monclient(hunting): _negotiate 2024-03-25T18:01:15.784 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.783+0000 7fe0708fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:15.784 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.783+0000 7fe0708fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:15.784 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.783+0000 7fe0708fd640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:01:15.784 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.783+0000 7fe0708fd640 1 -- 172.21.15.18:0/1254456252 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe058003520 con 0x7fe06c0a98f0 2024-03-25T18:01:15.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.789+0000 7fe0708fd640 1 -- 172.21.15.18:0/1254456252 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490101046 0 0) 0x7fe06c0b2a30 con 0x7fe06c1430c0 2024-03-25T18:01:15.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.789+0000 7fe0708fd640 10 monclient(hunting): _negotiate 2024-03-25T18:01:15.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.789+0000 7fe0708fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:15.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.789+0000 7fe0708fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:15.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.789+0000 7fe0708fd640 10 monclient(hunting): my global_id is 4451 2024-03-25T18:01:15.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.789+0000 7fe0708fd640 1 -- 172.21.15.18:0/1254456252 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe06c142520 con 0x7fe06c1430c0 2024-03-25T18:01:15.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.793+0000 7fe0708fd640 1 -- 172.21.15.18:0/1254456252 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911949931 0 0) 0x7fe06c0b28a0 con 0x7fe06c13f710 2024-03-25T18:01:15.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.793+0000 7fe0708fd640 10 monclient(hunting): _negotiate 2024-03-25T18:01:15.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.793+0000 7fe0708fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:15.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.793+0000 7fe0708fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:15.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.793+0000 7fe0708fd640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:01:15.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.793+0000 7fe0708fd640 1 -- 172.21.15.18:0/1254456252 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe06c0b2a30 con 0x7fe06c13f710 2024-03-25T18:01:15.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.795+0000 7fe0708fd640 1 -- 172.21.15.18:0/1254456252 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1575326415 0 0) 0x7fe058003520 con 0x7fe06c0a98f0 2024-03-25T18:01:15.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.795+0000 7fe0708fd640 1 -- 172.21.15.18:0/1254456252 >> v1:172.21.15.18:6790/0 conn(0x7fe06c1430c0 legacy=0x7fe06c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:15.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.795+0000 7fe0708fd640 1 -- 172.21.15.18:0/1254456252 >> v1:172.21.15.169:6789/0 conn(0x7fe06c13f710 legacy=0x7fe06c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:15.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.795+0000 7fe0708fd640 10 monclient: _finish_hunting 0 2024-03-25T18:01:15.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.795+0000 7fe0708fd640 1 monclient: found mon.noname-a 2024-03-25T18:01:15.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.795+0000 7fe0708fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:01:15.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.795+0000 7fe0708fd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:01:15.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.795+0000 7fe0708fd640 1 -- 172.21.15.18:0/1254456252 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06c0b44c0 con 0x7fe06c0a98f0 2024-03-25T18:01:15.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.795+0000 7fe0708fd640 10 monclient: _finish_auth 0 2024-03-25T18:01:15.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.795+0000 7fe0708fd640 10 monclient: _check_auth_tickets 2024-03-25T18:01:15.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.795+0000 7fe0708fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:01:15.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.795+0000 7fe0708fd640 1 -- 172.21.15.18:0/1254456252 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fe068001690 con 0x7fe06c0a98f0 2024-03-25T18:01:15.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.795+0000 7fe0708fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:15.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.795+0000 7fe0708fd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:01:15.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.795+0000 7fe0708fd640 10 monclient: dump: 2024-03-25T18:01:15.796 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:15.796 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:15.796 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:15.796 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:15.797 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:15.797 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:15.797 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:15.797 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:15.797 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:15.797 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:15.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.795+0000 7fe073b63640 5 monclient: authenticate success, global_id 4443 2024-03-25T18:01:15.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.795+0000 7fe073b63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:01:15.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.806+0000 7fe0708fd640 1 -- 172.21.15.18:0/1254456252 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0680017f0 con 0x7fe06c0a98f0 2024-03-25T18:01:15.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.806+0000 7fe0708fd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:01:15.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.806+0000 7fe0708fd640 1 -- 172.21.15.18:0/1254456252 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fe068001c00 con 0x7fe06c0a98f0 2024-03-25T18:01:15.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.806+0000 7fe073b63640 10 monclient: get_monmap_and_config success 2024-03-25T18:01:15.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.806+0000 7fe073b63640 4 set_mon_vals no callback set 2024-03-25T18:01:15.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.806+0000 7fe0708fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:15.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.806+0000 7fe0708fd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:01:15.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.806+0000 7fe0708fd640 10 monclient: dump: 2024-03-25T18:01:15.806 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:15.806 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:15.807 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:15.807 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:15.807 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:15.807 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:15.807 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:15.807 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:15.807 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:15.807 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:15.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.807+0000 7fe073b63640 10 monclient: shutdown 2024-03-25T18:01:15.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.807+0000 7fe073b63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:01:15.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.807+0000 7fe073b63640 1 -- 172.21.15.18:0/1254456252 >> v1:172.21.15.18:6789/0 conn(0x7fe06c0a98f0 legacy=0x7fe06c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:15.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.807+0000 7fe073b63640 1 -- 172.21.15.18:0/1254456252 shutdown_connections 2024-03-25T18:01:15.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.807+0000 7fe073b63640 1 -- 172.21.15.18:0/1254456252 >> 172.21.15.18:0/1254456252 conn(0x7fe06c0b7ea0 msgr2=0x7fe06c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:01:15.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.807+0000 7fe073b63640 1 -- 172.21.15.18:0/1254456252 shutdown_connections 2024-03-25T18:01:15.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.807+0000 7fe073b63640 1 -- 172.21.15.18:0/1254456252 wait complete. 2024-03-25T18:01:15.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.807+0000 7fe073b63640 10 monclient: build_initial_monmap 2024-03-25T18:01:15.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.807+0000 7fe073b63640 10 monclient: monmap: 2024-03-25T18:01:15.810 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:01:15.810 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:01:15.810 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:01:15.810 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:01:15.810 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:01:15.810 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:01:15.810 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:01:15.811 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:01:15.811 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:01:15.811 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:15.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.808+0000 7fe073b63640 1 librados: starting msgr at 2024-03-25T18:01:15.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.808+0000 7fe073b63640 1 librados: starting objecter 2024-03-25T18:01:15.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.808+0000 7fe073b63640 1 Processor -- start 2024-03-25T18:01:15.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.808+0000 7fe073b63640 1 -- start start 2024-03-25T18:01:15.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.808+0000 7fe073b63640 1 librados: setting wanted keys 2024-03-25T18:01:15.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.808+0000 7fe073b63640 1 librados: calling monclient init 2024-03-25T18:01:15.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.808+0000 7fe073b63640 10 monclient: init 2024-03-25T18:01:15.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.808+0000 7fe073b63640 10 monclient: _reopen_session rank -1 2024-03-25T18:01:15.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.808+0000 7fe073b63640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:01:15.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.808+0000 7fe073b63640 10 monclient(hunting): picked mon.noname-a con 0x7fe06c0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:01:15.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.808+0000 7fe073b63640 10 monclient(hunting): picked mon.noname-b con 0x7fe06c13f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:01:15.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.808+0000 7fe073b63640 10 monclient(hunting): picked mon.noname-c con 0x7fe06c1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:01:15.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.808+0000 7fe073b63640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe06c0b7ce0 con 0x7fe06c0a98f0 2024-03-25T18:01:15.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.808+0000 7fe073b63640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe06c1319b0 con 0x7fe06c13f710 2024-03-25T18:01:15.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.808+0000 7fe073b63640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe06c0b7800 con 0x7fe06c1430c0 2024-03-25T18:01:15.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.808+0000 7fe073b63640 10 monclient(hunting): _renew_subs 2024-03-25T18:01:15.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.808+0000 7fe073b63640 10 monclient(hunting): authenticate will time out at 7204.386541s 2024-03-25T18:01:15.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.815+0000 7fe072100640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fe06c1430c0 0x7fe06c1312a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:54594/0 (socket says 172.21.15.18:54594) 2024-03-25T18:01:15.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.815+0000 7fe072100640 1 -- 172.21.15.18:0/2588185581 learned_addr learned my addr 172.21.15.18:0/2588185581 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:01:15.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.816+0000 7fe0627fc640 10 client.?.objecter ms_handle_connect 0x7fe06c1430c0 2024-03-25T18:01:15.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.816+0000 7fe0627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:15.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.823+0000 7fe0627fc640 1 -- 172.21.15.18:0/2588185581 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960785354 0 0) 0x7fe06c0b7800 con 0x7fe06c1430c0 2024-03-25T18:01:15.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.823+0000 7fe0627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:01:15.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.823+0000 7fe0627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:15.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.823+0000 7fe0627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:15.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.823+0000 7fe0627fc640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:01:15.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.823+0000 7fe0627fc640 1 -- 172.21.15.18:0/2588185581 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe050004eb0 con 0x7fe06c1430c0 2024-03-25T18:01:15.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.824+0000 7fe0627fc640 10 client.?.objecter ms_handle_connect 0x7fe06c13f710 2024-03-25T18:01:15.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.824+0000 7fe0627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:15.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.831+0000 7fe0627fc640 10 client.?.objecter ms_handle_connect 0x7fe06c0a98f0 2024-03-25T18:01:15.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.831+0000 7fe0627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:15.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.831+0000 7fe0627fc640 1 -- 172.21.15.18:0/2588185581 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3974041441 0 0) 0x7fe050004eb0 con 0x7fe06c1430c0 2024-03-25T18:01:15.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.831+0000 7fe0627fc640 1 -- 172.21.15.18:0/2588185581 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe06c0b7800 con 0x7fe06c1430c0 2024-03-25T18:01:15.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.831+0000 7fe0627fc640 1 -- 172.21.15.18:0/2588185581 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fe064001c90 con 0x7fe06c1430c0 2024-03-25T18:01:15.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.831+0000 7fe0627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:15.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.831+0000 7fe0627fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:01:15.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.831+0000 7fe0627fc640 10 monclient(hunting): dump: 2024-03-25T18:01:15.832 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:15.832 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:15.832 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:15.832 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:15.832 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:15.832 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:15.832 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:15.832 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:15.832 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:15.832 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:15.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.831+0000 7fe0627fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:01:15.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.838+0000 7fe0627fc640 1 -- 172.21.15.18:0/2588185581 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544963191 0 0) 0x7fe06c1319b0 con 0x7fe06c13f710 2024-03-25T18:01:15.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.838+0000 7fe0627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:01:15.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.838+0000 7fe0627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:15.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.838+0000 7fe0627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:15.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.838+0000 7fe0627fc640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:01:15.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.838+0000 7fe0627fc640 1 -- 172.21.15.18:0/2588185581 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe050004eb0 con 0x7fe06c13f710 2024-03-25T18:01:15.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.838+0000 7fe0627fc640 1 -- 172.21.15.18:0/2588185581 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2623104763 0 0) 0x7fe06c0b7800 con 0x7fe06c1430c0 2024-03-25T18:01:15.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe0627fc640 1 -- 172.21.15.18:0/2588185581 >> v1:172.21.15.169:6789/0 conn(0x7fe06c13f710 legacy=0x7fe06c130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:15.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe0627fc640 1 -- 172.21.15.18:0/2588185581 >> v1:172.21.15.18:6789/0 conn(0x7fe06c0a98f0 legacy=0x7fe06c1dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:15.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe0627fc640 10 monclient: _finish_hunting 0 2024-03-25T18:01:15.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe0627fc640 1 monclient: found mon.c 2024-03-25T18:01:15.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe0627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:01:15.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe0627fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:01:15.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe0627fc640 1 -- 172.21.15.18:0/2588185581 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06c136f50 con 0x7fe06c1430c0 2024-03-25T18:01:15.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe0627fc640 10 monclient: _finish_auth 0 2024-03-25T18:01:15.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe0627fc640 10 monclient: _check_auth_tickets 2024-03-25T18:01:15.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe0627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:01:15.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe073b63640 5 monclient: authenticate success, global_id 4454 2024-03-25T18:01:15.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe073b63640 10 monclient: _renew_subs 2024-03-25T18:01:15.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe073b63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:01:15.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe073b63640 1 -- 172.21.15.18:0/2588185581 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe06c1350f0 con 0x7fe06c1430c0 2024-03-25T18:01:15.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe073b63640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:01:15.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe073b63640 10 monclient: _renew_subs 2024-03-25T18:01:15.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe073b63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:01:15.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe073b63640 1 -- 172.21.15.18:0/2588185581 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe06c135c60 con 0x7fe06c1430c0 2024-03-25T18:01:15.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.839+0000 7fe073b63640 1 librados: init done 2024-03-25T18:01:15.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.840+0000 7fe073b63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:15.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.840+0000 7fe073b63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:15.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.840+0000 7fe073b63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:01:15.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:15.840+0000 7fe073b63640 1 -- 172.21.15.18:0/2588185581 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe06c0627b0 con 0x7fe06c1430c0 2024-03-25T18:01:16.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.068+0000 7fe0627fc640 1 -- 172.21.15.18:0/2588185581 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe064003970 con 0x7fe06c1430c0 2024-03-25T18:01:16.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.068+0000 7fe0627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:01:16.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.068+0000 7fe063fff640 4 set_mon_vals no callback set 2024-03-25T18:01:16.149 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.149+0000 7fe0627fc640 1 -- 172.21.15.18:0/2588185581 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fe064003d40 con 0x7fe06c1430c0 2024-03-25T18:01:16.149 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.149+0000 7fe0627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:16.149 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.149+0000 7fe0627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:01:16.149 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.149+0000 7fe0627fc640 10 monclient: dump: 2024-03-25T18:01:16.149 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:16.150 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:16.150 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:16.150 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:16.150 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:16.150 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:16.150 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:16.150 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:16.150 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:16.150 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:16.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.150+0000 7fe0627fc640 1 -- 172.21.15.18:0/2588185581 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fe064003ea0 con 0x7fe06c1430c0 2024-03-25T18:01:16.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.150+0000 7fe0718ff640 10 monclient: get_auth_request con 0x7fe05006dd90 auth_method 0 2024-03-25T18:01:16.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.151+0000 7fe0627fc640 10 client.4454.objecter ms_handle_connect 0x7fe05006dd90 2024-03-25T18:01:16.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.156+0000 7fe0627fc640 1 -- 172.21.15.18:0/2588185581 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fe064054b00 con 0x7fe06c1430c0 2024-03-25T18:01:16.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.156+0000 7fe0627fc640 10 client.4454.objecter ms_dispatch 0x7fe06c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:01:16.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.156+0000 7fe0627fc640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:01:16.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.156+0000 7fe0627fc640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:01:16.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.157+0000 7fe0627fc640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-25T18:01:16.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.157+0000 7fe0627fc640 1 -- 172.21.15.18:0/2588185581 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe064050610 con 0x7fe06c1430c0 2024-03-25T18:01:16.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.157+0000 7fe0627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:16.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.157+0000 7fe0627fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:01:16.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.398+0000 7fe073b63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:16.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.398+0000 7fe073b63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:16.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.398+0000 7fe073b63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:01:16.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.398+0000 7fe073b63640 1 -- 172.21.15.18:0/2588185581 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe06c13e270 con 0x7fe06c1430c0 2024-03-25T18:01:16.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.424+0000 7fe0627fc640 1 -- 172.21.15.18:0/2588185581 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fe064052540 con 0x7fe06c1430c0 2024-03-25T18:01:16.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.424+0000 7fe0627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:16.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.424+0000 7fe0627fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:01:16.424 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:01:16.425 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:01:16.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.427+0000 7fe073b63640 10 librados: watch_flush enter 2024-03-25T18:01:16.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.427+0000 7fe073b63640 10 librados: watch_flush exit 2024-03-25T18:01:16.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.427+0000 7fe073b63640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-25T18:01:16.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.427+0000 7fe073b63640 10 client.4454.objecter successfully canceled tick 2024-03-25T18:01:16.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.427+0000 7fe073b63640 1 -- 172.21.15.18:0/2588185581 >> v1:172.21.15.18:6800/3721244451 conn(0x7fe05006dd90 legacy=0x7fe050070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:16.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.427+0000 7fe073b63640 10 monclient: shutdown 2024-03-25T18:01:16.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.427+0000 7fe073b63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:01:16.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.427+0000 7fe073b63640 1 -- 172.21.15.18:0/2588185581 >> v1:172.21.15.18:6790/0 conn(0x7fe06c1430c0 legacy=0x7fe06c1312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:16.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.427+0000 7fe073b63640 1 -- 172.21.15.18:0/2588185581 shutdown_connections 2024-03-25T18:01:16.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.427+0000 7fe073b63640 1 -- 172.21.15.18:0/2588185581 >> 172.21.15.18:0/2588185581 conn(0x7fe06c0b7ea0 msgr2=0x7fe06c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:01:16.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.427+0000 7fe073b63640 1 -- 172.21.15.18:0/2588185581 shutdown_connections 2024-03-25T18:01:16.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.427+0000 7fe073b63640 1 -- 172.21.15.18:0/2588185581 wait complete. 2024-03-25T18:01:16.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:16.427+0000 7fe073b63640 1 librados: shutdown 2024-03-25T18:01:16.439 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:01:22.933 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:20.725 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:22.938 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:20.731 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:22.977 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:20.770 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:22.992 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:20.785 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:22.995 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:20.744 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:23.001 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:20.750 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:23.004 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:20.797 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:23.036 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:20.829 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:23.036 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:20.785 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:23.245 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:01:23.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.367+0000 7f5b7b7fe640 10 monclient: get_monmap_and_config 2024-03-25T18:01:23.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.367+0000 7f5b7b7fe640 10 monclient: build_initial_monmap 2024-03-25T18:01:23.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.367+0000 7f5b7b7fe640 10 monclient: monmap: 2024-03-25T18:01:23.368 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:01:23.368 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:23.368 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:01:23.367963+0000 2024-03-25T18:01:23.368 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:01:23.367963+0000 2024-03-25T18:01:23.368 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:01:23.368 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:23.368 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:01:23.369 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:01:23.369 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:01:23.369 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:23.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.368+0000 7f5b7b7fe640 1 Processor -- start 2024-03-25T18:01:23.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.368+0000 7f5b7b7fe640 1 -- start start 2024-03-25T18:01:23.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.368+0000 7f5b7b7fe640 10 monclient: init 2024-03-25T18:01:23.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.369+0000 7f5b7b7fe640 10 monclient: _reopen_session rank -1 2024-03-25T18:01:23.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.369+0000 7f5b7b7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:01:23.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.369+0000 7f5b7b7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5b740a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:01:23.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.369+0000 7f5b7b7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5b7413f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:01:23.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.369+0000 7f5b7b7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5b741430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:01:23.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.369+0000 7f5b7b7fe640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b74142520 con 0x7f5b7413f710 2024-03-25T18:01:23.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.369+0000 7f5b7b7fe640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b740b28a0 con 0x7f5b740a98f0 2024-03-25T18:01:23.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.369+0000 7f5b7b7fe640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b740b2a30 con 0x7f5b741430c0 2024-03-25T18:01:23.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.369+0000 7f5b7b7fe640 10 monclient(hunting): _renew_subs 2024-03-25T18:01:23.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.369+0000 7f5b7b7fe640 10 monclient(hunting): authenticate will time out at 7211.947216s 2024-03-25T18:01:23.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.382+0000 7f5b7959a640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f5b740a98f0 0x7f5b740a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:56636/0 (socket says 172.21.15.18:56636) 2024-03-25T18:01:23.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.383+0000 7f5b7959a640 1 -- 172.21.15.18:0/938004815 learned_addr learned my addr 172.21.15.18:0/938004815 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:01:23.410 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.409+0000 7f5b6bfff640 1 -- 172.21.15.18:0/938004815 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242933296 0 0) 0x7f5b74142520 con 0x7f5b7413f710 2024-03-25T18:01:23.410 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.409+0000 7f5b6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:01:23.410 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.409+0000 7f5b6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:23.410 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.409+0000 7f5b6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:23.410 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.409+0000 7f5b6bfff640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:01:23.410 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.409+0000 7f5b6bfff640 1 -- 172.21.15.18:0/938004815 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b60003520 con 0x7f5b7413f710 2024-03-25T18:01:23.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.413+0000 7f5b6bfff640 1 -- 172.21.15.18:0/938004815 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287457131 0 0) 0x7f5b740b28a0 con 0x7f5b740a98f0 2024-03-25T18:01:23.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.413+0000 7f5b6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:01:23.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.413+0000 7f5b6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:23.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.413+0000 7f5b6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:23.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.413+0000 7f5b6bfff640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:01:23.413 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.413+0000 7f5b6bfff640 1 -- 172.21.15.18:0/938004815 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b74142520 con 0x7f5b740a98f0 2024-03-25T18:01:23.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.416+0000 7f5b6bfff640 1 -- 172.21.15.18:0/938004815 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392282935 0 0) 0x7f5b740b2a30 con 0x7f5b741430c0 2024-03-25T18:01:23.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.416+0000 7f5b6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:01:23.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.416+0000 7f5b6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:23.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.416+0000 7f5b6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:23.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.416+0000 7f5b6bfff640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:01:23.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.416+0000 7f5b6bfff640 1 -- 172.21.15.18:0/938004815 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b740b28a0 con 0x7f5b741430c0 2024-03-25T18:01:23.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.429+0000 7f5b6bfff640 1 -- 172.21.15.18:0/938004815 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2865867385 0 0) 0x7f5b60003520 con 0x7f5b7413f710 2024-03-25T18:01:23.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.429+0000 7f5b6bfff640 1 -- 172.21.15.18:0/938004815 >> v1:172.21.15.18:6790/0 conn(0x7f5b741430c0 legacy=0x7f5b7414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:23.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.429+0000 7f5b6bfff640 1 -- 172.21.15.18:0/938004815 >> v1:172.21.15.169:6789/0 conn(0x7f5b740a98f0 legacy=0x7f5b740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:23.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.429+0000 7f5b6bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:01:23.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.429+0000 7f5b6bfff640 1 monclient: found mon.noname-a 2024-03-25T18:01:23.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.429+0000 7f5b6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:01:23.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.429+0000 7f5b6bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:01:23.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.429+0000 7f5b6bfff640 1 -- 172.21.15.18:0/938004815 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b740b44c0 con 0x7f5b7413f710 2024-03-25T18:01:23.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.429+0000 7f5b6bfff640 10 monclient: _finish_auth 0 2024-03-25T18:01:23.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.429+0000 7f5b6bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:01:23.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.429+0000 7f5b6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:01:23.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.429+0000 7f5b6bfff640 1 -- 172.21.15.18:0/938004815 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5b64001690 con 0x7f5b7413f710 2024-03-25T18:01:23.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.429+0000 7f5b6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:23.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.429+0000 7f5b6bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:01:23.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.429+0000 7f5b6bfff640 10 monclient: dump: 2024-03-25T18:01:23.430 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:23.430 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:23.431 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:23.431 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:23.431 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:23.431 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:23.431 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:23.431 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:23.431 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:23.431 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:23.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.429+0000 7f5b7b7fe640 5 monclient: authenticate success, global_id 4449 2024-03-25T18:01:23.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.429+0000 7f5b7b7fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:01:23.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.456+0000 7f5b6bfff640 1 -- 172.21.15.18:0/938004815 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b640017f0 con 0x7f5b7413f710 2024-03-25T18:01:23.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.456+0000 7f5b6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:01:23.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.456+0000 7f5b6bfff640 1 -- 172.21.15.18:0/938004815 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5b64001c00 con 0x7f5b7413f710 2024-03-25T18:01:23.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.456+0000 7f5b6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:23.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.456+0000 7f5b6bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:01:23.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.456+0000 7f5b6bfff640 10 monclient: dump: 2024-03-25T18:01:23.457 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:23.457 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:23.457 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:23.457 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:23.457 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:23.457 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:23.457 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:23.457 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:23.457 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:23.457 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:23.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.456+0000 7f5b7b7fe640 10 monclient: get_monmap_and_config success 2024-03-25T18:01:23.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.456+0000 7f5b7b7fe640 4 set_mon_vals no callback set 2024-03-25T18:01:23.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.457+0000 7f5b7b7fe640 10 monclient: shutdown 2024-03-25T18:01:23.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.457+0000 7f5b7b7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:01:23.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.457+0000 7f5b7b7fe640 1 -- 172.21.15.18:0/938004815 >> v1:172.21.15.18:6789/0 conn(0x7f5b7413f710 legacy=0x7f5b741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:23.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.457+0000 7f5b7b7fe640 1 -- 172.21.15.18:0/938004815 shutdown_connections 2024-03-25T18:01:23.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.457+0000 7f5b7b7fe640 1 -- 172.21.15.18:0/938004815 >> 172.21.15.18:0/938004815 conn(0x7f5b740b7ea0 msgr2=0x7f5b740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:01:23.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.457+0000 7f5b7b7fe640 1 -- 172.21.15.18:0/938004815 shutdown_connections 2024-03-25T18:01:23.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.457+0000 7f5b7b7fe640 1 -- 172.21.15.18:0/938004815 wait complete. 2024-03-25T18:01:23.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.458+0000 7f5b7b7fe640 10 monclient: build_initial_monmap 2024-03-25T18:01:23.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.458+0000 7f5b7b7fe640 10 monclient: monmap: 2024-03-25T18:01:23.460 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:01:23.460 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:01:23.460 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:01:23.460 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:01:23.461 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:01:23.461 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:01:23.461 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:01:23.461 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:01:23.461 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:01:23.461 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:23.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.458+0000 7f5b7b7fe640 1 librados: starting msgr at 2024-03-25T18:01:23.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.458+0000 7f5b7b7fe640 1 librados: starting objecter 2024-03-25T18:01:23.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.458+0000 7f5b7b7fe640 1 Processor -- start 2024-03-25T18:01:23.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.458+0000 7f5b7b7fe640 1 -- start start 2024-03-25T18:01:23.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.458+0000 7f5b7b7fe640 1 librados: setting wanted keys 2024-03-25T18:01:23.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.458+0000 7f5b7b7fe640 1 librados: calling monclient init 2024-03-25T18:01:23.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.458+0000 7f5b7b7fe640 10 monclient: init 2024-03-25T18:01:23.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.459+0000 7f5b7b7fe640 10 monclient: _reopen_session rank -1 2024-03-25T18:01:23.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.459+0000 7f5b7b7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:01:23.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.459+0000 7f5b7b7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5b740a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:01:23.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.459+0000 7f5b7b7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f5b7413f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:01:23.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.459+0000 7f5b7b7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5b741430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:01:23.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.459+0000 7f5b7b7fe640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b741397c0 con 0x7f5b740a98f0 2024-03-25T18:01:23.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.459+0000 7f5b7b7fe640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b741daa20 con 0x7f5b7413f710 2024-03-25T18:01:23.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.459+0000 7f5b7b7fe640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b741d8210 con 0x7f5b741430c0 2024-03-25T18:01:23.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.459+0000 7f5b7b7fe640 10 monclient(hunting): _renew_subs 2024-03-25T18:01:23.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.459+0000 7f5b7b7fe640 10 monclient(hunting): authenticate will time out at 7212.037366s 2024-03-25T18:01:23.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.471+0000 7f5b7959a640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f5b740a98f0 0x7f5b741daf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:51932/0 (socket says 172.21.15.18:51932) 2024-03-25T18:01:23.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.471+0000 7f5b7959a640 1 -- 172.21.15.18:0/4085618386 learned_addr learned my addr 172.21.15.18:0/4085618386 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:01:23.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.473+0000 7f5b69ffb640 10 client.?.objecter ms_handle_connect 0x7f5b740a98f0 2024-03-25T18:01:23.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.473+0000 7f5b69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:23.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.473+0000 7f5b69ffb640 10 client.?.objecter ms_handle_connect 0x7f5b7413f710 2024-03-25T18:01:23.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.473+0000 7f5b69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:23.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.477+0000 7f5b69ffb640 10 client.?.objecter ms_handle_connect 0x7f5b741430c0 2024-03-25T18:01:23.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.477+0000 7f5b69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:23.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.478+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140299563 0 0) 0x7f5b741397c0 con 0x7f5b740a98f0 2024-03-25T18:01:23.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.478+0000 7f5b69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:01:23.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.478+0000 7f5b69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:23.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.478+0000 7f5b69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:23.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.478+0000 7f5b69ffb640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:01:23.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.478+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b58004eb0 con 0x7f5b740a98f0 2024-03-25T18:01:23.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.484+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764629515 0 0) 0x7f5b741d8210 con 0x7f5b741430c0 2024-03-25T18:01:23.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.485+0000 7f5b69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:01:23.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.485+0000 7f5b69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:23.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.485+0000 7f5b69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:23.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.485+0000 7f5b69ffb640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:01:23.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.485+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b741397c0 con 0x7f5b741430c0 2024-03-25T18:01:23.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.486+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2987623045 0 0) 0x7f5b741daa20 con 0x7f5b7413f710 2024-03-25T18:01:23.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.486+0000 7f5b69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:01:23.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.486+0000 7f5b69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:23.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.486+0000 7f5b69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:23.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.486+0000 7f5b69ffb640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:01:23.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.486+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b741d8210 con 0x7f5b7413f710 2024-03-25T18:01:23.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.492+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 762162944 0 0) 0x7f5b741397c0 con 0x7f5b741430c0 2024-03-25T18:01:23.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.493+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b741daa20 con 0x7f5b741430c0 2024-03-25T18:01:23.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.493+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5b6c001b50 con 0x7f5b741430c0 2024-03-25T18:01:23.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.493+0000 7f5b69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:23.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.493+0000 7f5b69ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:01:23.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.493+0000 7f5b69ffb640 10 monclient(hunting): dump: 2024-03-25T18:01:23.494 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:23.494 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:23.494 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:23.494 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:23.494 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:23.494 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:23.494 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:23.494 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:23.494 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:23.494 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:23.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.493+0000 7f5b69ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:01:23.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.493+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3769767322 0 0) 0x7f5b741d8210 con 0x7f5b7413f710 2024-03-25T18:01:23.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.493+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b741397c0 con 0x7f5b7413f710 2024-03-25T18:01:23.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.493+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5b64001b00 con 0x7f5b7413f710 2024-03-25T18:01:23.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.493+0000 7f5b69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:23.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.493+0000 7f5b69ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:01:23.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.493+0000 7f5b69ffb640 10 monclient(hunting): dump: 2024-03-25T18:01:23.495 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:23.495 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:23.495 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:23.495 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:23.495 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:23.495 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:23.495 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:23.495 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:23.496 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:23.496 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:23.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.501+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3813897726 0 0) 0x7f5b58004eb0 con 0x7f5b740a98f0 2024-03-25T18:01:23.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.501+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b741d8210 con 0x7f5b740a98f0 2024-03-25T18:01:23.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.501+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5b700019f0 con 0x7f5b740a98f0 2024-03-25T18:01:23.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.501+0000 7f5b69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:23.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.501+0000 7f5b69ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:01:23.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.501+0000 7f5b69ffb640 10 monclient(hunting): dump: 2024-03-25T18:01:23.502 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:23.502 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:23.502 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:23.502 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:23.502 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:23.502 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:23.502 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:23.502 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:23.502 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:23.503 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:23.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1640337597 0 0) 0x7f5b741397c0 con 0x7f5b7413f710 2024-03-25T18:01:23.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 >> v1:172.21.15.18:6790/0 conn(0x7f5b741430c0 legacy=0x7f5b741d7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:23.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 >> v1:172.21.15.18:6789/0 conn(0x7f5b740a98f0 legacy=0x7f5b741daf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:23.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b69ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:01:23.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b69ffb640 1 monclient: found mon.b 2024-03-25T18:01:23.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:01:23.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b69ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:23.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b741db650 con 0x7f5b7413f710 2024-03-25T18:01:23.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b69ffb640 10 monclient: _finish_auth 0 2024-03-25T18:01:23.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b69ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:01:23.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:01:23.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b7b7fe640 5 monclient: authenticate success, global_id 4441 2024-03-25T18:01:23.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b7b7fe640 10 monclient: _renew_subs 2024-03-25T18:01:23.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b7b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:23.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b7b7fe640 1 -- 172.21.15.18:0/4085618386 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b74137720 con 0x7f5b7413f710 2024-03-25T18:01:23.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b7b7fe640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:01:23.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b7b7fe640 10 monclient: _renew_subs 2024-03-25T18:01:23.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b7b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:23.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b7b7fe640 1 -- 172.21.15.18:0/4085618386 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b74137d50 con 0x7f5b7413f710 2024-03-25T18:01:23.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.506+0000 7f5b7b7fe640 1 librados: init done 2024-03-25T18:01:23.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.507+0000 7f5b7b7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:23.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.507+0000 7f5b7b7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:23.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.508+0000 7f5b7b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:23.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.508+0000 7f5b7b7fe640 1 -- 172.21.15.18:0/4085618386 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b741db840 con 0x7f5b7413f710 2024-03-25T18:01:23.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.791+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b640010a0 con 0x7f5b7413f710 2024-03-25T18:01:23.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.791+0000 7f5b69ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:01:23.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.791+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5b640038a0 con 0x7f5b7413f710 2024-03-25T18:01:23.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.791+0000 7f5b69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:23.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.791+0000 7f5b69ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:01:23.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.791+0000 7f5b69ffb640 10 monclient: dump: 2024-03-25T18:01:23.792 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:23.792 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:23.792 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:23.792 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:23.792 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:23.792 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:23.792 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:23.792 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:23.793 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:23.793 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:23.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.791+0000 7f5b6b7fe640 4 set_mon_vals no callback set 2024-03-25T18:01:23.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.792+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f5b64019870 con 0x7f5b7413f710 2024-03-25T18:01:23.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.792+0000 7f5b7959a640 10 monclient: get_auth_request con 0x7f5b5806e160 auth_method 0 2024-03-25T18:01:23.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.792+0000 7f5b69ffb640 10 client.4441.objecter ms_handle_connect 0x7f5b5806e160 2024-03-25T18:01:23.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.798+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f5b640533c0 con 0x7f5b7413f710 2024-03-25T18:01:23.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.798+0000 7f5b69ffb640 10 client.4441.objecter ms_dispatch 0x7f5b74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:01:23.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.798+0000 7f5b69ffb640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:01:23.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.798+0000 7f5b69ffb640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:01:23.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.798+0000 7f5b69ffb640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-25T18:01:23.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.799+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b6404ed00 con 0x7f5b7413f710 2024-03-25T18:01:23.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.799+0000 7f5b69ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:23.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:23.799+0000 7f5b69ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:01:24.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.025+0000 7f5b7b7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:24.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.025+0000 7f5b7b7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:24.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.025+0000 7f5b7b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:24.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.025+0000 7f5b7b7fe640 1 -- 172.21.15.18:0/4085618386 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b74074cc0 con 0x7f5b7413f710 2024-03-25T18:01:24.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.052+0000 7f5b69ffb640 1 -- 172.21.15.18:0/4085618386 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f5b64001210 con 0x7f5b7413f710 2024-03-25T18:01:24.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.052+0000 7f5b69ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:24.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.052+0000 7f5b69ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:01:24.053 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:01:24.053 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:01:24.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.054+0000 7f5b7b7fe640 10 librados: watch_flush enter 2024-03-25T18:01:24.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.054+0000 7f5b7b7fe640 10 librados: watch_flush exit 2024-03-25T18:01:24.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.054+0000 7f5b7b7fe640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-25T18:01:24.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.054+0000 7f5b7b7fe640 10 client.4441.objecter successfully canceled tick 2024-03-25T18:01:24.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.054+0000 7f5b7b7fe640 1 -- 172.21.15.18:0/4085618386 >> v1:172.21.15.18:6800/3721244451 conn(0x7f5b5806e160 legacy=0x7f5b58070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:24.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.055+0000 7f5b7b7fe640 10 monclient: shutdown 2024-03-25T18:01:24.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.055+0000 7f5b7b7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:01:24.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.055+0000 7f5b7b7fe640 1 -- 172.21.15.18:0/4085618386 >> v1:172.21.15.169:6789/0 conn(0x7f5b7413f710 legacy=0x7f5b741d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:24.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.055+0000 7f5b7b7fe640 1 -- 172.21.15.18:0/4085618386 shutdown_connections 2024-03-25T18:01:24.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.055+0000 7f5b7b7fe640 1 -- 172.21.15.18:0/4085618386 >> 172.21.15.18:0/4085618386 conn(0x7f5b740b7ea0 msgr2=0x7f5b7413c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:01:24.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.055+0000 7f5b7b7fe640 1 -- 172.21.15.18:0/4085618386 shutdown_connections 2024-03-25T18:01:24.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.055+0000 7f5b7b7fe640 1 -- 172.21.15.18:0/4085618386 wait complete. 2024-03-25T18:01:24.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:24.055+0000 7f5b7b7fe640 1 librados: shutdown 2024-03-25T18:01:24.067 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:01:30.874 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:01:30.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:30.989+0000 7fcea8538640 10 monclient: get_monmap_and_config 2024-03-25T18:01:30.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:30.989+0000 7fcea8538640 10 monclient: build_initial_monmap 2024-03-25T18:01:30.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:30.989+0000 7fcea8538640 10 monclient: monmap: 2024-03-25T18:01:30.989 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:01:30.990 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:30.990 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:01:30.989726+0000 2024-03-25T18:01:30.990 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:01:30.989726+0000 2024-03-25T18:01:30.990 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:01:30.990 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:30.990 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:01:30.990 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:01:30.990 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:01:30.990 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:30.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:30.990+0000 7fcea8538640 1 Processor -- start 2024-03-25T18:01:30.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:30.990+0000 7fcea8538640 1 -- start start 2024-03-25T18:01:30.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:30.990+0000 7fcea8538640 10 monclient: init 2024-03-25T18:01:30.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:30.991+0000 7fcea8538640 10 monclient: _reopen_session rank -1 2024-03-25T18:01:30.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:30.991+0000 7fcea8538640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:01:30.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:30.991+0000 7fcea8538640 10 monclient(hunting): picked mon.noname-c con 0x7fcea00a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:01:30.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:30.991+0000 7fcea8538640 10 monclient(hunting): picked mon.noname-b con 0x7fcea013f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:01:30.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:30.991+0000 7fcea8538640 10 monclient(hunting): picked mon.noname-a con 0x7fcea01430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:01:30.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:30.991+0000 7fcea8538640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea0142520 con 0x7fcea01430c0 2024-03-25T18:01:30.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:30.991+0000 7fcea8538640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea00b28a0 con 0x7fcea00a98f0 2024-03-25T18:01:30.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:30.991+0000 7fcea8538640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea00b2a30 con 0x7fcea013f710 2024-03-25T18:01:30.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:30.991+0000 7fcea8538640 10 monclient(hunting): _renew_subs 2024-03-25T18:01:30.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:30.991+0000 7fcea8538640 10 monclient(hunting): authenticate will time out at 7219.569186s 2024-03-25T18:01:31.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.002+0000 7fcea62d4640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fcea00a98f0 0x7fcea00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:43218/0 (socket says 172.21.15.18:43218) 2024-03-25T18:01:31.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.002+0000 7fcea62d4640 1 -- 172.21.15.18:0/667058315 learned_addr learned my addr 172.21.15.18:0/667058315 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:01:31.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.016+0000 7fcea52d2640 1 -- 172.21.15.18:0/667058315 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230931335 0 0) 0x7fcea00b2a30 con 0x7fcea013f710 2024-03-25T18:01:31.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.016+0000 7fcea52d2640 10 monclient(hunting): _negotiate 2024-03-25T18:01:31.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.016+0000 7fcea52d2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:31.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.016+0000 7fcea52d2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:31.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.016+0000 7fcea52d2640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:01:31.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.016+0000 7fcea52d2640 1 -- 172.21.15.18:0/667058315 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce8c003520 con 0x7fcea013f710 2024-03-25T18:01:31.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.018+0000 7fcea52d2640 1 -- 172.21.15.18:0/667058315 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667076434 0 0) 0x7fcea00b28a0 con 0x7fcea00a98f0 2024-03-25T18:01:31.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.018+0000 7fcea52d2640 10 monclient(hunting): _negotiate 2024-03-25T18:01:31.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.018+0000 7fcea52d2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:31.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.018+0000 7fcea52d2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:31.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.018+0000 7fcea52d2640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:01:31.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.018+0000 7fcea52d2640 1 -- 172.21.15.18:0/667058315 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea00b2a30 con 0x7fcea00a98f0 2024-03-25T18:01:31.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.026+0000 7fcea52d2640 1 -- 172.21.15.18:0/667058315 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157303568 0 0) 0x7fcea0142520 con 0x7fcea01430c0 2024-03-25T18:01:31.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.026+0000 7fcea52d2640 10 monclient(hunting): _negotiate 2024-03-25T18:01:31.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.026+0000 7fcea52d2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:31.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.026+0000 7fcea52d2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:31.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.026+0000 7fcea52d2640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:01:31.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.026+0000 7fcea52d2640 1 -- 172.21.15.18:0/667058315 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea00b28a0 con 0x7fcea01430c0 2024-03-25T18:01:31.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea52d2640 1 -- 172.21.15.18:0/667058315 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 691940854 0 0) 0x7fcea00b2a30 con 0x7fcea00a98f0 2024-03-25T18:01:31.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea52d2640 1 -- 172.21.15.18:0/667058315 >> v1:172.21.15.18:6790/0 conn(0x7fcea013f710 legacy=0x7fcea01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:31.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea52d2640 1 -- 172.21.15.18:0/667058315 >> v1:172.21.15.18:6789/0 conn(0x7fcea01430c0 legacy=0x7fcea014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:31.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea52d2640 10 monclient: _finish_hunting 0 2024-03-25T18:01:31.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea52d2640 1 monclient: found mon.noname-c 2024-03-25T18:01:31.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea52d2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:01:31.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea52d2640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:01:31.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea52d2640 1 -- 172.21.15.18:0/667058315 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea00b44c0 con 0x7fcea00a98f0 2024-03-25T18:01:31.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea52d2640 10 monclient: _finish_auth 0 2024-03-25T18:01:31.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea52d2640 10 monclient: _check_auth_tickets 2024-03-25T18:01:31.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea52d2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:01:31.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea52d2640 1 -- 172.21.15.18:0/667058315 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fce9c001670 con 0x7fcea00a98f0 2024-03-25T18:01:31.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea8538640 5 monclient: authenticate success, global_id 4444 2024-03-25T18:01:31.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea8538640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:01:31.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea52d2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:31.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea52d2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:01:31.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea52d2640 10 monclient: dump: 2024-03-25T18:01:31.034 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:31.034 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:31.034 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:31.034 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:31.034 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:31.034 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:31.034 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:31.034 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:31.034 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:31.034 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:31.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.032+0000 7fcea8538640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:01:31.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.053+0000 7fcea52d2640 1 -- 172.21.15.18:0/667058315 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce9c0017d0 con 0x7fcea00a98f0 2024-03-25T18:01:31.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.053+0000 7fcea52d2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:01:31.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.053+0000 7fcea52d2640 1 -- 172.21.15.18:0/667058315 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fce9c001be0 con 0x7fcea00a98f0 2024-03-25T18:01:31.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.053+0000 7fcea8538640 10 monclient: get_monmap_and_config success 2024-03-25T18:01:31.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.053+0000 7fcea8538640 4 set_mon_vals no callback set 2024-03-25T18:01:31.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.053+0000 7fcea52d2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:31.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.053+0000 7fcea52d2640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:01:31.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.053+0000 7fcea52d2640 10 monclient: dump: 2024-03-25T18:01:31.053 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:31.054 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:31.054 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:31.054 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:31.054 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:31.054 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:31.054 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:31.054 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:31.054 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:31.054 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:31.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.053+0000 7fcea8538640 10 monclient: shutdown 2024-03-25T18:01:31.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.053+0000 7fcea8538640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:01:31.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.053+0000 7fcea8538640 1 -- 172.21.15.18:0/667058315 >> v1:172.21.15.169:6789/0 conn(0x7fcea00a98f0 legacy=0x7fcea00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:31.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.054+0000 7fcea8538640 1 -- 172.21.15.18:0/667058315 shutdown_connections 2024-03-25T18:01:31.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.054+0000 7fcea8538640 1 -- 172.21.15.18:0/667058315 >> 172.21.15.18:0/667058315 conn(0x7fcea00b7ea0 msgr2=0x7fcea00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:01:31.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.054+0000 7fcea8538640 1 -- 172.21.15.18:0/667058315 shutdown_connections 2024-03-25T18:01:31.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.054+0000 7fcea8538640 1 -- 172.21.15.18:0/667058315 wait complete. 2024-03-25T18:01:31.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.054+0000 7fcea8538640 10 monclient: build_initial_monmap 2024-03-25T18:01:31.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.054+0000 7fcea8538640 10 monclient: monmap: 2024-03-25T18:01:31.057 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:01:31.057 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:01:31.057 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:01:31.057 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:01:31.057 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:01:31.057 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:01:31.058 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:01:31.058 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:01:31.058 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:01:31.058 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:31.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.055+0000 7fcea8538640 1 librados: starting msgr at 2024-03-25T18:01:31.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.055+0000 7fcea8538640 1 librados: starting objecter 2024-03-25T18:01:31.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.055+0000 7fcea8538640 1 Processor -- start 2024-03-25T18:01:31.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.055+0000 7fcea8538640 1 -- start start 2024-03-25T18:01:31.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.055+0000 7fcea8538640 1 librados: setting wanted keys 2024-03-25T18:01:31.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.055+0000 7fcea8538640 1 librados: calling monclient init 2024-03-25T18:01:31.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.055+0000 7fcea8538640 10 monclient: init 2024-03-25T18:01:31.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.055+0000 7fcea8538640 10 monclient: _reopen_session rank -1 2024-03-25T18:01:31.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.056+0000 7fcea8538640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:01:31.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.056+0000 7fcea8538640 10 monclient(hunting): picked mon.noname-b con 0x7fcea00a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:01:31.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.056+0000 7fcea8538640 10 monclient(hunting): picked mon.noname-c con 0x7fcea013f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:01:31.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.056+0000 7fcea8538640 10 monclient(hunting): picked mon.noname-a con 0x7fcea01430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:01:31.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.056+0000 7fcea8538640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea013b180 con 0x7fcea01430c0 2024-03-25T18:01:31.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.056+0000 7fcea8538640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea01db890 con 0x7fcea00a98f0 2024-03-25T18:01:31.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.056+0000 7fcea8538640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea00b7b90 con 0x7fcea013f710 2024-03-25T18:01:31.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.056+0000 7fcea8538640 10 monclient(hunting): _renew_subs 2024-03-25T18:01:31.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.056+0000 7fcea8538640 10 monclient(hunting): authenticate will time out at 7219.633993s 2024-03-25T18:01:31.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.062+0000 7fcea5ad3640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fcea013f710 0x7fcea01daa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:54914/0 (socket says 172.21.15.18:54914) 2024-03-25T18:01:31.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.062+0000 7fcea6ad5640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fcea01430c0 0x7fcea01db180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:48610/0 (socket says 172.21.15.18:48610) 2024-03-25T18:01:31.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.062+0000 7fcea5ad3640 1 -- 172.21.15.18:0/3210099431 learned_addr learned my addr 172.21.15.18:0/3210099431 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:01:31.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.063+0000 7fce96ffd640 10 client.?.objecter ms_handle_connect 0x7fcea01430c0 2024-03-25T18:01:31.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.063+0000 7fce96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:31.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.065+0000 7fce96ffd640 10 client.?.objecter ms_handle_connect 0x7fcea013f710 2024-03-25T18:01:31.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.065+0000 7fce96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:31.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.065+0000 7fce96ffd640 10 client.?.objecter ms_handle_connect 0x7fcea00a98f0 2024-03-25T18:01:31.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.065+0000 7fce96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:31.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.074+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325601931 0 0) 0x7fcea013b180 con 0x7fcea01430c0 2024-03-25T18:01:31.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.074+0000 7fce96ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:01:31.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.074+0000 7fce96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:31.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.074+0000 7fce96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:31.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.074+0000 7fce96ffd640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:01:31.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.074+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce84004eb0 con 0x7fcea01430c0 2024-03-25T18:01:31.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.082+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559349366 0 0) 0x7fcea01db890 con 0x7fcea00a98f0 2024-03-25T18:01:31.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.082+0000 7fce96ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:01:31.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.082+0000 7fce96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:31.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.082+0000 7fce96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:31.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.082+0000 7fce96ffd640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:01:31.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.082+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea013b180 con 0x7fcea00a98f0 2024-03-25T18:01:31.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.085+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049904565 0 0) 0x7fcea00b7b90 con 0x7fcea013f710 2024-03-25T18:01:31.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.085+0000 7fce96ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:01:31.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.085+0000 7fce96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:31.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.085+0000 7fce96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:31.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.085+0000 7fce96ffd640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:01:31.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.085+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea01db890 con 0x7fcea013f710 2024-03-25T18:01:31.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.101+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1202171394 0 0) 0x7fcea013b180 con 0x7fcea00a98f0 2024-03-25T18:01:31.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.101+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcea00b7b90 con 0x7fcea00a98f0 2024-03-25T18:01:31.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.101+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fce9c001ae0 con 0x7fcea00a98f0 2024-03-25T18:01:31.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.101+0000 7fce96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:31.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.101+0000 7fce96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:01:31.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.101+0000 7fce96ffd640 10 monclient(hunting): dump: 2024-03-25T18:01:31.102 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:31.102 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:31.102 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:31.102 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:31.102 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:31.102 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:31.102 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:31.102 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:31.102 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:31.102 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:31.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.101+0000 7fce96ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:01:31.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.102+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 326729599 0 0) 0x7fcea01db890 con 0x7fcea013f710 2024-03-25T18:01:31.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.102+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcea013b180 con 0x7fcea013f710 2024-03-25T18:01:31.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.102+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fce90001910 con 0x7fcea013f710 2024-03-25T18:01:31.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.102+0000 7fce96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:31.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.102+0000 7fce96ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:01:31.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.102+0000 7fce96ffd640 10 monclient(hunting): dump: 2024-03-25T18:01:31.104 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:31.104 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:31.104 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:31.104 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:31.104 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:31.104 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:31.104 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:31.104 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:31.104 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:31.104 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:31.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2017227466 0 0) 0x7fce84004eb0 con 0x7fcea01430c0 2024-03-25T18:01:31.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcea01db890 con 0x7fcea01430c0 2024-03-25T18:01:31.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fce98001bd0 con 0x7fcea01430c0 2024-03-25T18:01:31.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fce96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:31.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fce96ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:01:31.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fce96ffd640 10 monclient(hunting): dump: 2024-03-25T18:01:31.112 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:31.112 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:31.112 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:31.113 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:31.113 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:31.113 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:31.113 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:31.113 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:31.113 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:31.113 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:31.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2690252201 0 0) 0x7fcea00b7b90 con 0x7fcea00a98f0 2024-03-25T18:01:31.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 >> v1:172.21.15.18:6790/0 conn(0x7fcea013f710 legacy=0x7fcea01daa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:31.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 >> v1:172.21.15.18:6789/0 conn(0x7fcea01430c0 legacy=0x7fcea01db180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:31.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fce96ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:01:31.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fce96ffd640 1 monclient: found mon.b 2024-03-25T18:01:31.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fce96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:01:31.113 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fce96ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:31.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea013b3f0 con 0x7fcea00a98f0 2024-03-25T18:01:31.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fce96ffd640 10 monclient: _finish_auth 0 2024-03-25T18:01:31.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fce96ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:01:31.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fce96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:01:31.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fcea8538640 5 monclient: authenticate success, global_id 4447 2024-03-25T18:01:31.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fcea8538640 10 monclient: _renew_subs 2024-03-25T18:01:31.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fcea8538640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:31.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fcea8538640 1 -- 172.21.15.18:0/3210099431 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea013a1b0 con 0x7fcea00a98f0 2024-03-25T18:01:31.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fcea8538640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:01:31.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fcea8538640 10 monclient: _renew_subs 2024-03-25T18:01:31.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fcea8538640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:31.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fcea8538640 1 -- 172.21.15.18:0/3210099431 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea013b9f0 con 0x7fcea00a98f0 2024-03-25T18:01:31.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.111+0000 7fcea8538640 1 librados: init done 2024-03-25T18:01:31.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.112+0000 7fcea8538640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:31.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.112+0000 7fcea8538640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:31.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.112+0000 7fcea8538640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:31.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.112+0000 7fcea8538640 1 -- 172.21.15.18:0/3210099431 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea00b3ef0 con 0x7fcea00a98f0 2024-03-25T18:01:31.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.399+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce9c001210 con 0x7fcea00a98f0 2024-03-25T18:01:31.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.399+0000 7fce96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:01:31.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.399+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fce9c004c30 con 0x7fcea00a98f0 2024-03-25T18:01:31.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.399+0000 7fce96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:31.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.399+0000 7fce96ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:01:31.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.399+0000 7fce96ffd640 10 monclient: dump: 2024-03-25T18:01:31.400 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:31.400 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:31.400 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:31.400 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:31.400 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:31.400 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:31.400 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:31.400 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:31.400 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:31.400 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:31.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.399+0000 7fcea4ad1640 4 set_mon_vals no callback set 2024-03-25T18:01:31.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.400+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fce9c01ac40 con 0x7fcea00a98f0 2024-03-25T18:01:31.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.401+0000 7fcea5ad3640 10 monclient: get_auth_request con 0x7fce8406e0f0 auth_method 0 2024-03-25T18:01:31.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.401+0000 7fce96ffd640 10 client.4447.objecter ms_handle_connect 0x7fce8406e0f0 2024-03-25T18:01:31.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.403+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fce9c054860 con 0x7fcea00a98f0 2024-03-25T18:01:31.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.403+0000 7fce96ffd640 10 client.4447.objecter ms_dispatch 0x7fcea0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:01:31.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.403+0000 7fce96ffd640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:01:31.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.403+0000 7fce96ffd640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:01:31.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.403+0000 7fce96ffd640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-25T18:01:31.406 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.406+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce9c050370 con 0x7fcea00a98f0 2024-03-25T18:01:31.406 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.406+0000 7fce96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:31.406 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.406+0000 7fce96ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:01:31.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.639+0000 7fcea8538640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:31.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.639+0000 7fcea8538640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:31.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.639+0000 7fcea8538640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:31.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.639+0000 7fcea8538640 1 -- 172.21.15.18:0/3210099431 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcea00627b0 con 0x7fcea00a98f0 2024-03-25T18:01:31.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.666+0000 7fce96ffd640 1 -- 172.21.15.18:0/3210099431 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fce9c0522a0 con 0x7fcea00a98f0 2024-03-25T18:01:31.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.666+0000 7fce96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:31.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.666+0000 7fce96ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:01:31.667 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:01:31.667 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:01:31.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.668+0000 7fcea8538640 10 librados: watch_flush enter 2024-03-25T18:01:31.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.669+0000 7fcea8538640 10 librados: watch_flush exit 2024-03-25T18:01:31.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.669+0000 7fcea8538640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-25T18:01:31.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.669+0000 7fcea8538640 10 client.4447.objecter successfully canceled tick 2024-03-25T18:01:31.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.669+0000 7fcea8538640 1 -- 172.21.15.18:0/3210099431 >> v1:172.21.15.18:6800/3721244451 conn(0x7fce8406e0f0 legacy=0x7fce840705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:31.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.669+0000 7fcea8538640 10 monclient: shutdown 2024-03-25T18:01:31.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.669+0000 7fcea8538640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:01:31.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.669+0000 7fcea8538640 1 -- 172.21.15.18:0/3210099431 >> v1:172.21.15.169:6789/0 conn(0x7fcea00a98f0 legacy=0x7fcea00b7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:31.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.669+0000 7fcea8538640 1 -- 172.21.15.18:0/3210099431 shutdown_connections 2024-03-25T18:01:31.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.669+0000 7fcea8538640 1 -- 172.21.15.18:0/3210099431 >> 172.21.15.18:0/3210099431 conn(0x7fcea00b7ea0 msgr2=0x7fcea0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:01:31.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.669+0000 7fcea8538640 1 -- 172.21.15.18:0/3210099431 shutdown_connections 2024-03-25T18:01:31.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.669+0000 7fcea8538640 1 -- 172.21.15.18:0/3210099431 wait complete. 2024-03-25T18:01:31.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:31.669+0000 7fcea8538640 1 librados: shutdown 2024-03-25T18:01:31.681 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:01:38.487 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:01:38.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.607+0000 7f352cd7b640 10 monclient: get_monmap_and_config 2024-03-25T18:01:38.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.607+0000 7f352cd7b640 10 monclient: build_initial_monmap 2024-03-25T18:01:38.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.607+0000 7f352cd7b640 10 monclient: monmap: 2024-03-25T18:01:38.608 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:01:38.608 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:38.608 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:01:38.608531+0000 2024-03-25T18:01:38.608 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:01:38.608531+0000 2024-03-25T18:01:38.609 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:01:38.609 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:38.609 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:01:38.609 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:01:38.609 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:01:38.609 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:38.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.607+0000 7f352cd7b640 1 Processor -- start 2024-03-25T18:01:38.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.607+0000 7f352cd7b640 1 -- start start 2024-03-25T18:01:38.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.607+0000 7f352cd7b640 10 monclient: init 2024-03-25T18:01:38.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.608+0000 7f352cd7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:01:38.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.608+0000 7f352cd7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:01:38.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.608+0000 7f352cd7b640 10 monclient(hunting): picked mon.noname-b con 0x7f35280b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T18:01:38.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.608+0000 7f352cd7b640 10 monclient(hunting): picked mon.noname-c con 0x7f35280b1730 addr v1:172.21.15.169:6789/0 2024-03-25T18:01:38.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.608+0000 7f352cd7b640 10 monclient(hunting): picked mon.noname-a con 0x7f352814be90 addr v1:172.21.15.18:6789/0 2024-03-25T18:01:38.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.608+0000 7f352cd7b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35280af940 con 0x7f352814be90 2024-03-25T18:01:38.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.608+0000 7f352cd7b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35280b0a70 con 0x7f35280b1730 2024-03-25T18:01:38.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.608+0000 7f352cd7b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35280b0d50 con 0x7f35280b4d60 2024-03-25T18:01:38.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.608+0000 7f352cd7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:01:38.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.608+0000 7f352cd7b640 10 monclient(hunting): authenticate will time out at 7227.187331s 2024-03-25T18:01:38.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.623+0000 7f351dd9b640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f35280b1730 0x7f35280b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:42114/0 (socket says 172.21.15.18:42114) 2024-03-25T18:01:38.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.623+0000 7f351dd9b640 1 -- 172.21.15.18:0/3938439416 learned_addr learned my addr 172.21.15.18:0/3938439416 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:01:38.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.642+0000 7f3525d9b640 1 -- 172.21.15.18:0/3938439416 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188646831 0 0) 0x7f35280b0a70 con 0x7f35280b1730 2024-03-25T18:01:38.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.642+0000 7f3525d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:01:38.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.642+0000 7f3525d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:38.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.642+0000 7f3525d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:38.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.642+0000 7f3525d9b640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:01:38.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.642+0000 7f3525d9b640 1 -- 172.21.15.18:0/3938439416 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3514003520 con 0x7f35280b1730 2024-03-25T18:01:38.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.655+0000 7f3525d9b640 1 -- 172.21.15.18:0/3938439416 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824987876 0 0) 0x7f35280b0d50 con 0x7f35280b4d60 2024-03-25T18:01:38.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.655+0000 7f3525d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:01:38.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.655+0000 7f3525d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:38.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.655+0000 7f3525d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:38.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.655+0000 7f3525d9b640 10 monclient(hunting): my global_id is 4469 2024-03-25T18:01:38.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.655+0000 7f3525d9b640 1 -- 172.21.15.18:0/3938439416 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35280b0a70 con 0x7f35280b4d60 2024-03-25T18:01:38.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.656+0000 7f3525d9b640 1 -- 172.21.15.18:0/3938439416 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1103694725 0 0) 0x7f3514003520 con 0x7f35280b1730 2024-03-25T18:01:38.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.656+0000 7f3525d9b640 1 -- 172.21.15.18:0/3938439416 >> v1:172.21.15.18:6790/0 conn(0x7f35280b4d60 legacy=0x7f352813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:38.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.656+0000 7f3525d9b640 1 -- 172.21.15.18:0/3938439416 >> v1:172.21.15.18:6789/0 conn(0x7f352814be90 legacy=0x7f352814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:38.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.656+0000 7f3525d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:01:38.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.656+0000 7f3525d9b640 1 monclient: found mon.noname-c 2024-03-25T18:01:38.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.656+0000 7f3525d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:01:38.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.656+0000 7f3525d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:01:38.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.656+0000 7f3525d9b640 1 -- 172.21.15.18:0/3938439416 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35280b1030 con 0x7f35280b1730 2024-03-25T18:01:38.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.656+0000 7f3525d9b640 10 monclient: _finish_auth 0 2024-03-25T18:01:38.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.656+0000 7f3525d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:01:38.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.656+0000 7f3525d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:01:38.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.656+0000 7f3525d9b640 1 -- 172.21.15.18:0/3938439416 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3510001690 con 0x7f35280b1730 2024-03-25T18:01:38.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.656+0000 7f3525d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:38.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.656+0000 7f3525d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:01:38.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.656+0000 7f3525d9b640 10 monclient: dump: 2024-03-25T18:01:38.659 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:38.659 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:38.659 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:38.659 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:38.659 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:38.659 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:38.659 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:38.659 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:38.659 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:38.660 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:38.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.656+0000 7f352cd7b640 5 monclient: authenticate success, global_id 4450 2024-03-25T18:01:38.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.656+0000 7f352cd7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:01:38.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.668+0000 7f3525d9b640 1 -- 172.21.15.18:0/3938439416 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35100017f0 con 0x7f35280b1730 2024-03-25T18:01:38.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.668+0000 7f3525d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:01:38.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.668+0000 7f3525d9b640 1 -- 172.21.15.18:0/3938439416 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3510001c00 con 0x7f35280b1730 2024-03-25T18:01:38.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.668+0000 7f352cd7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:01:38.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.668+0000 7f352cd7b640 4 set_mon_vals no callback set 2024-03-25T18:01:38.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.668+0000 7f3525d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:38.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.668+0000 7f3525d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:01:38.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.668+0000 7f3525d9b640 10 monclient: dump: 2024-03-25T18:01:38.670 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:38.670 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:38.670 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:38.670 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:38.670 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:38.670 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:38.670 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:38.670 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:38.671 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:38.671 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:38.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.669+0000 7f352cd7b640 10 monclient: shutdown 2024-03-25T18:01:38.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.669+0000 7f352cd7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:01:38.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.669+0000 7f352cd7b640 1 -- 172.21.15.18:0/3938439416 >> v1:172.21.15.169:6789/0 conn(0x7f35280b1730 legacy=0x7f35280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:38.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.669+0000 7f352cd7b640 1 -- 172.21.15.18:0/3938439416 shutdown_connections 2024-03-25T18:01:38.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.669+0000 7f352cd7b640 1 -- 172.21.15.18:0/3938439416 >> 172.21.15.18:0/3938439416 conn(0x7f35280617a0 msgr2=0x7f3528061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:01:38.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.669+0000 7f352cd7b640 1 -- 172.21.15.18:0/3938439416 shutdown_connections 2024-03-25T18:01:38.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.669+0000 7f352cd7b640 1 -- 172.21.15.18:0/3938439416 wait complete. 2024-03-25T18:01:38.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.670+0000 7f352cd7b640 10 monclient: build_initial_monmap 2024-03-25T18:01:38.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.670+0000 7f352cd7b640 10 monclient: monmap: 2024-03-25T18:01:38.673 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:01:38.673 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:01:38.673 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:01:38.673 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:01:38.673 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:01:38.673 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:01:38.673 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:01:38.673 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:01:38.673 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:01:38.674 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:38.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.671+0000 7f352cd7b640 1 librados: starting msgr at 2024-03-25T18:01:38.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.671+0000 7f352cd7b640 1 librados: starting objecter 2024-03-25T18:01:38.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.671+0000 7f352cd7b640 1 Processor -- start 2024-03-25T18:01:38.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.671+0000 7f352cd7b640 1 -- start start 2024-03-25T18:01:38.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.671+0000 7f352cd7b640 1 librados: setting wanted keys 2024-03-25T18:01:38.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.671+0000 7f352cd7b640 1 librados: calling monclient init 2024-03-25T18:01:38.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.671+0000 7f352cd7b640 10 monclient: init 2024-03-25T18:01:38.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.672+0000 7f352cd7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:01:38.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.672+0000 7f352cd7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:01:38.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.672+0000 7f352cd7b640 10 monclient(hunting): picked mon.noname-c con 0x7f35280b1730 addr v1:172.21.15.18:6790/0 2024-03-25T18:01:38.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.672+0000 7f352cd7b640 10 monclient(hunting): picked mon.noname-b con 0x7f35280b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T18:01:38.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.672+0000 7f352cd7b640 10 monclient(hunting): picked mon.noname-a con 0x7f352814be90 addr v1:172.21.15.18:6789/0 2024-03-25T18:01:38.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.672+0000 7f352cd7b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35281db590 con 0x7f352814be90 2024-03-25T18:01:38.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.672+0000 7f352cd7b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3528141fc0 con 0x7f35280b4d60 2024-03-25T18:01:38.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.672+0000 7f352cd7b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3528140590 con 0x7f35280b1730 2024-03-25T18:01:38.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.672+0000 7f352cd7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:01:38.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.672+0000 7f352cd7b640 10 monclient(hunting): authenticate will time out at 7227.251511s 2024-03-25T18:01:38.687 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.687+0000 7f351dd9b640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f35280b4d60 0x7f3528134e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:42142/0 (socket says 172.21.15.18:42142) 2024-03-25T18:01:38.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.687+0000 7f351dd9b640 1 -- 172.21.15.18:0/3303299440 learned_addr learned my addr 172.21.15.18:0/3303299440 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:01:38.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.688+0000 7f351f7fe640 10 client.?.objecter ms_handle_connect 0x7f35280b4d60 2024-03-25T18:01:38.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.688+0000 7f351f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:38.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.693+0000 7f351f7fe640 10 client.?.objecter ms_handle_connect 0x7f352814be90 2024-03-25T18:01:38.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.693+0000 7f351f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:38.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.695+0000 7f351f7fe640 10 client.?.objecter ms_handle_connect 0x7f35280b1730 2024-03-25T18:01:38.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.695+0000 7f351f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:38.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.699+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458965170 0 0) 0x7f3528141fc0 con 0x7f35280b4d60 2024-03-25T18:01:38.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.699+0000 7f351f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:01:38.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.699+0000 7f351f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:38.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.699+0000 7f351f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:38.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.699+0000 7f351f7fe640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:01:38.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.699+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3508004eb0 con 0x7f35280b4d60 2024-03-25T18:01:38.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.706+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498323947 0 0) 0x7f35281db590 con 0x7f352814be90 2024-03-25T18:01:38.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.706+0000 7f351f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:01:38.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.706+0000 7f351f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:38.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.706+0000 7f351f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:38.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.706+0000 7f351f7fe640 10 monclient(hunting): my global_id is 4464 2024-03-25T18:01:38.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.706+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3528141fc0 con 0x7f352814be90 2024-03-25T18:01:38.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.708+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711848437 0 0) 0x7f3528140590 con 0x7f35280b1730 2024-03-25T18:01:38.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.708+0000 7f351f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:01:38.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.708+0000 7f351f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:38.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.708+0000 7f351f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:38.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.708+0000 7f351f7fe640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:01:38.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.708+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35281db590 con 0x7f35280b1730 2024-03-25T18:01:38.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.712+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 752460839 0 0) 0x7f3508004eb0 con 0x7f35280b4d60 2024-03-25T18:01:38.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.712+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3528140590 con 0x7f35280b4d60 2024-03-25T18:01:38.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.712+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3510001c40 con 0x7f35280b4d60 2024-03-25T18:01:38.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.712+0000 7f351f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:38.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.712+0000 7f351f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:01:38.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.712+0000 7f351f7fe640 10 monclient(hunting): dump: 2024-03-25T18:01:38.713 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:38.713 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:38.713 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:38.713 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:38.713 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:38.713 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:38.714 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:38.714 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:38.714 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:38.714 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:38.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.712+0000 7f351f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:01:38.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.723+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3357260406 0 0) 0x7f3528140590 con 0x7f35280b4d60 2024-03-25T18:01:38.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.723+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 >> v1:172.21.15.18:6790/0 conn(0x7f35280b1730 legacy=0x7f35281dae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:38.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.723+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 >> v1:172.21.15.18:6789/0 conn(0x7f352814be90 legacy=0x7f35281e3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:38.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.723+0000 7f351f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:01:38.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.723+0000 7f351f7fe640 1 monclient: found mon.b 2024-03-25T18:01:38.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.723+0000 7f351f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:01:38.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.724+0000 7f351f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:38.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.724+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35281d5440 con 0x7f35280b4d60 2024-03-25T18:01:38.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.724+0000 7f351f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:01:38.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.724+0000 7f351f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:01:38.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.724+0000 7f351f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:01:38.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.724+0000 7f352cd7b640 5 monclient: authenticate success, global_id 4453 2024-03-25T18:01:38.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.724+0000 7f352cd7b640 10 monclient: _renew_subs 2024-03-25T18:01:38.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.724+0000 7f352cd7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:38.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.724+0000 7f352cd7b640 1 -- 172.21.15.18:0/3303299440 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3528135550 con 0x7f35280b4d60 2024-03-25T18:01:38.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.724+0000 7f352cd7b640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:01:38.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.724+0000 7f352cd7b640 10 monclient: _renew_subs 2024-03-25T18:01:38.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.724+0000 7f352cd7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:38.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.724+0000 7f352cd7b640 1 -- 172.21.15.18:0/3303299440 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3528137c40 con 0x7f35280b4d60 2024-03-25T18:01:38.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.724+0000 7f352cd7b640 1 librados: init done 2024-03-25T18:01:38.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.725+0000 7f352cd7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:38.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.725+0000 7f352cd7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:38.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.725+0000 7f352cd7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:38.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:38.725+0000 7f352cd7b640 1 -- 172.21.15.18:0/3303299440 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3528140720 con 0x7f35280b4d60 2024-03-25T18:01:39.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.002+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3510004a90 con 0x7f35280b4d60 2024-03-25T18:01:39.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.002+0000 7f351f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:01:39.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.002+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3510004de0 con 0x7f35280b4d60 2024-03-25T18:01:39.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.002+0000 7f351f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:39.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.002+0000 7f351f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:01:39.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.002+0000 7f351f7fe640 10 monclient: dump: 2024-03-25T18:01:39.004 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:39.004 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:39.005 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:39.005 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:39.005 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:39.005 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:39.005 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:39.005 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:39.005 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:39.005 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:39.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.002+0000 7f352559a640 4 set_mon_vals no callback set 2024-03-25T18:01:39.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.003+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f3510005000 con 0x7f35280b4d60 2024-03-25T18:01:39.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.004+0000 7f352659c640 10 monclient: get_auth_request con 0x7f350806ddf0 auth_method 0 2024-03-25T18:01:39.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.004+0000 7f351f7fe640 10 client.4453.objecter ms_handle_connect 0x7f350806ddf0 2024-03-25T18:01:39.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.008+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f35100549b0 con 0x7f35280b4d60 2024-03-25T18:01:39.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.008+0000 7f351f7fe640 10 client.4453.objecter ms_dispatch 0x7f3528000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:01:39.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.008+0000 7f351f7fe640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:01:39.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.008+0000 7f351f7fe640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:01:39.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.008+0000 7f351f7fe640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-25T18:01:39.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.009+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35100504c0 con 0x7f35280b4d60 2024-03-25T18:01:39.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.009+0000 7f351f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:39.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.009+0000 7f351f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:01:39.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.226+0000 7f352cd7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:39.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.226+0000 7f352cd7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:39.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.227+0000 7f352cd7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:01:39.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.227+0000 7f352cd7b640 1 -- 172.21.15.18:0/3303299440 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35280627b0 con 0x7f35280b4d60 2024-03-25T18:01:39.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.377+0000 7f351f7fe640 1 -- 172.21.15.18:0/3303299440 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f35100523f0 con 0x7f35280b4d60 2024-03-25T18:01:39.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.377+0000 7f351f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:39.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.377+0000 7f351f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:01:39.379 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:01:39.379 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:01:39.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.379+0000 7f352cd7b640 10 librados: watch_flush enter 2024-03-25T18:01:39.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.379+0000 7f352cd7b640 10 librados: watch_flush exit 2024-03-25T18:01:39.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.379+0000 7f352cd7b640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-25T18:01:39.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.379+0000 7f352cd7b640 10 client.4453.objecter successfully canceled tick 2024-03-25T18:01:39.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.379+0000 7f352cd7b640 1 -- 172.21.15.18:0/3303299440 >> v1:172.21.15.18:6800/3721244451 conn(0x7f350806ddf0 legacy=0x7f35080702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:39.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.379+0000 7f352cd7b640 10 monclient: shutdown 2024-03-25T18:01:39.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.379+0000 7f352cd7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:01:39.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.379+0000 7f352cd7b640 1 -- 172.21.15.18:0/3303299440 >> v1:172.21.15.169:6789/0 conn(0x7f35280b4d60 legacy=0x7f3528134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:39.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.379+0000 7f352cd7b640 1 -- 172.21.15.18:0/3303299440 shutdown_connections 2024-03-25T18:01:39.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.380+0000 7f352cd7b640 1 -- 172.21.15.18:0/3303299440 >> 172.21.15.18:0/3303299440 conn(0x7f35280617a0 msgr2=0x7f352813c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:01:39.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.380+0000 7f352cd7b640 1 -- 172.21.15.18:0/3303299440 shutdown_connections 2024-03-25T18:01:39.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.380+0000 7f352cd7b640 1 -- 172.21.15.18:0/3303299440 wait complete. 2024-03-25T18:01:39.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:39.380+0000 7f352cd7b640 1 librados: shutdown 2024-03-25T18:01:39.391 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:01:42.972 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:40.764 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:42.982 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:40.775 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:43.031 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:40.779 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:43.032 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:05:40.825 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:43.035 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:40.784 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:43.038 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:40.831 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:43.051 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:40.844 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:43.058 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:05:40.807 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:43.088 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:05:40.880 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:01:46.197 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:01:46.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.313+0000 7f8890c6a640 10 monclient: get_monmap_and_config 2024-03-25T18:01:46.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.313+0000 7f8890c6a640 10 monclient: build_initial_monmap 2024-03-25T18:01:46.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.313+0000 7f8890c6a640 10 monclient: monmap: 2024-03-25T18:01:46.314 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:01:46.315 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:46.315 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:01:46.314531+0000 2024-03-25T18:01:46.315 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:01:46.314531+0000 2024-03-25T18:01:46.315 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:01:46.315 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:46.315 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:01:46.315 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:01:46.315 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:01:46.315 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:46.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.314+0000 7f8890c6a640 1 Processor -- start 2024-03-25T18:01:46.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.314+0000 7f8890c6a640 1 -- start start 2024-03-25T18:01:46.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.314+0000 7f8890c6a640 10 monclient: init 2024-03-25T18:01:46.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.314+0000 7f8890c6a640 10 monclient: _reopen_session rank -1 2024-03-25T18:01:46.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.314+0000 7f8890c6a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:01:46.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.314+0000 7f8890c6a640 10 monclient(hunting): picked mon.noname-a con 0x7f888c0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:01:46.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.314+0000 7f8890c6a640 10 monclient(hunting): picked mon.noname-b con 0x7f888c13f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:01:46.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.314+0000 7f8890c6a640 10 monclient(hunting): picked mon.noname-c con 0x7f888c1430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:01:46.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.314+0000 7f8890c6a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888c142520 con 0x7f888c0a98f0 2024-03-25T18:01:46.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.314+0000 7f8890c6a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888c0b28a0 con 0x7f888c1430c0 2024-03-25T18:01:46.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.314+0000 7f8890c6a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888c0b2a30 con 0x7f888c13f710 2024-03-25T18:01:46.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.314+0000 7f8890c6a640 10 monclient(hunting): _renew_subs 2024-03-25T18:01:46.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.314+0000 7f8890c6a640 10 monclient(hunting): authenticate will time out at 7234.893401s 2024-03-25T18:01:46.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.322+0000 7f8883fff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f888c13f710 0x7f888c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:35294/0 (socket says 172.21.15.18:35294) 2024-03-25T18:01:46.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.322+0000 7f8883fff640 1 -- 172.21.15.18:0/3428408989 learned_addr learned my addr 172.21.15.18:0/3428408989 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:01:46.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.345+0000 7f8889d9b640 1 -- 172.21.15.18:0/3428408989 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999784604 0 0) 0x7f888c0b2a30 con 0x7f888c13f710 2024-03-25T18:01:46.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.345+0000 7f8889d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:01:46.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.345+0000 7f8889d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:46.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.345+0000 7f8889d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:46.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.345+0000 7f8889d9b640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:01:46.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.345+0000 7f8889d9b640 1 -- 172.21.15.18:0/3428408989 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8878003520 con 0x7f888c13f710 2024-03-25T18:01:46.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.350+0000 7f8889d9b640 1 -- 172.21.15.18:0/3428408989 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898466545 0 0) 0x7f888c0b28a0 con 0x7f888c1430c0 2024-03-25T18:01:46.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.350+0000 7f8889d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:01:46.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.350+0000 7f8889d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:46.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.350+0000 7f8889d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:46.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.350+0000 7f8889d9b640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:01:46.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.350+0000 7f8889d9b640 1 -- 172.21.15.18:0/3428408989 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f888c0b2a30 con 0x7f888c1430c0 2024-03-25T18:01:46.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.351+0000 7f8889d9b640 1 -- 172.21.15.18:0/3428408989 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570359160 0 0) 0x7f888c142520 con 0x7f888c0a98f0 2024-03-25T18:01:46.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.351+0000 7f8889d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:01:46.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.351+0000 7f8889d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:46.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.351+0000 7f8889d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:46.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.351+0000 7f8889d9b640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:01:46.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.351+0000 7f8889d9b640 1 -- 172.21.15.18:0/3428408989 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f888c0b28a0 con 0x7f888c0a98f0 2024-03-25T18:01:46.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.353+0000 7f8889d9b640 1 -- 172.21.15.18:0/3428408989 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2233395139 0 0) 0x7f8878003520 con 0x7f888c13f710 2024-03-25T18:01:46.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.353+0000 7f8889d9b640 1 -- 172.21.15.18:0/3428408989 >> v1:172.21.15.169:6789/0 conn(0x7f888c1430c0 legacy=0x7f888c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:46.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.354+0000 7f8889d9b640 1 -- 172.21.15.18:0/3428408989 >> v1:172.21.15.18:6789/0 conn(0x7f888c0a98f0 legacy=0x7f888c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:46.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.354+0000 7f8889d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:01:46.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.354+0000 7f8889d9b640 1 monclient: found mon.noname-b 2024-03-25T18:01:46.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.354+0000 7f8889d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:01:46.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.354+0000 7f8889d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:01:46.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.354+0000 7f8889d9b640 1 -- 172.21.15.18:0/3428408989 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888c0b44c0 con 0x7f888c13f710 2024-03-25T18:01:46.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.354+0000 7f8889d9b640 10 monclient: _finish_auth 0 2024-03-25T18:01:46.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.354+0000 7f8889d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:01:46.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.354+0000 7f8889d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:01:46.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.354+0000 7f8889d9b640 1 -- 172.21.15.18:0/3428408989 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8874001690 con 0x7f888c13f710 2024-03-25T18:01:46.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.354+0000 7f8889d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:46.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.354+0000 7f8889d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:01:46.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.354+0000 7f8889d9b640 10 monclient: dump: 2024-03-25T18:01:46.356 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:46.356 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:46.356 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:46.356 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:46.356 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:46.356 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:46.357 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:46.357 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:46.357 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:46.357 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:46.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.354+0000 7f8890c6a640 5 monclient: authenticate success, global_id 4475 2024-03-25T18:01:46.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.354+0000 7f8890c6a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:01:46.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.365+0000 7f8889d9b640 1 -- 172.21.15.18:0/3428408989 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88740017f0 con 0x7f888c13f710 2024-03-25T18:01:46.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.365+0000 7f8889d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:01:46.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.365+0000 7f8889d9b640 1 -- 172.21.15.18:0/3428408989 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8874001c00 con 0x7f888c13f710 2024-03-25T18:01:46.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.365+0000 7f8889d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:46.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.365+0000 7f8889d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:01:46.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.365+0000 7f8889d9b640 10 monclient: dump: 2024-03-25T18:01:46.367 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:46.367 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:46.367 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:46.367 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:46.367 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:46.367 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:46.368 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:46.368 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:46.368 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:46.368 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:46.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.365+0000 7f8890c6a640 10 monclient: get_monmap_and_config success 2024-03-25T18:01:46.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.365+0000 7f8890c6a640 4 set_mon_vals no callback set 2024-03-25T18:01:46.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.366+0000 7f8890c6a640 10 monclient: shutdown 2024-03-25T18:01:46.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.366+0000 7f8890c6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:01:46.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.366+0000 7f8890c6a640 1 -- 172.21.15.18:0/3428408989 >> v1:172.21.15.18:6790/0 conn(0x7f888c13f710 legacy=0x7f888c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:46.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.366+0000 7f8890c6a640 1 -- 172.21.15.18:0/3428408989 shutdown_connections 2024-03-25T18:01:46.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.366+0000 7f8890c6a640 1 -- 172.21.15.18:0/3428408989 >> 172.21.15.18:0/3428408989 conn(0x7f888c0b7ea0 msgr2=0x7f888c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:01:46.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.366+0000 7f8890c6a640 1 -- 172.21.15.18:0/3428408989 shutdown_connections 2024-03-25T18:01:46.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.366+0000 7f8890c6a640 1 -- 172.21.15.18:0/3428408989 wait complete. 2024-03-25T18:01:46.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.368+0000 7f8890c6a640 10 monclient: build_initial_monmap 2024-03-25T18:01:46.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.368+0000 7f8890c6a640 10 monclient: monmap: 2024-03-25T18:01:46.371 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:01:46.371 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:01:46.371 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:01:46.371 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:01:46.371 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:01:46.371 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:01:46.371 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:01:46.371 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:01:46.371 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:01:46.372 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:46.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.368+0000 7f8890c6a640 1 librados: starting msgr at 2024-03-25T18:01:46.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.368+0000 7f8890c6a640 1 librados: starting objecter 2024-03-25T18:01:46.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.368+0000 7f8890c6a640 1 Processor -- start 2024-03-25T18:01:46.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.368+0000 7f8890c6a640 1 -- start start 2024-03-25T18:01:46.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.368+0000 7f8890c6a640 1 librados: setting wanted keys 2024-03-25T18:01:46.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.368+0000 7f8890c6a640 1 librados: calling monclient init 2024-03-25T18:01:46.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.368+0000 7f8890c6a640 10 monclient: init 2024-03-25T18:01:46.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.369+0000 7f8890c6a640 10 monclient: _reopen_session rank -1 2024-03-25T18:01:46.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.369+0000 7f8890c6a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:01:46.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.369+0000 7f8890c6a640 10 monclient(hunting): picked mon.noname-a con 0x7f888c0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:01:46.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.369+0000 7f8890c6a640 10 monclient(hunting): picked mon.noname-c con 0x7f888c13f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:01:46.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.369+0000 7f8890c6a640 10 monclient(hunting): picked mon.noname-b con 0x7f888c1430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:01:46.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.369+0000 7f8890c6a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888c0ab8b0 con 0x7f888c0a98f0 2024-03-25T18:01:46.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.369+0000 7f8890c6a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888c0abb90 con 0x7f888c1430c0 2024-03-25T18:01:46.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.369+0000 7f8890c6a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888c137870 con 0x7f888c13f710 2024-03-25T18:01:46.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.369+0000 7f8890c6a640 10 monclient(hunting): _renew_subs 2024-03-25T18:01:46.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.369+0000 7f8890c6a640 10 monclient(hunting): authenticate will time out at 7234.947978s 2024-03-25T18:01:46.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.385+0000 7f888ad9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f888c1430c0 0x7f888c1e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:37196/0 (socket says 172.21.15.18:37196) 2024-03-25T18:01:46.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.385+0000 7f888ad9d640 1 -- 172.21.15.18:0/2855802642 learned_addr learned my addr 172.21.15.18:0/2855802642 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:01:46.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.388+0000 7f8882ffd640 10 client.?.objecter ms_handle_connect 0x7f888c1430c0 2024-03-25T18:01:46.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.388+0000 7f8882ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:46.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.389+0000 7f8882ffd640 10 client.?.objecter ms_handle_connect 0x7f888c13f710 2024-03-25T18:01:46.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.389+0000 7f8882ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:46.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.390+0000 7f8882ffd640 10 client.?.objecter ms_handle_connect 0x7f888c0a98f0 2024-03-25T18:01:46.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.390+0000 7f8882ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:46.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.396+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893678786 0 0) 0x7f888c137870 con 0x7f888c13f710 2024-03-25T18:01:46.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.396+0000 7f8882ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:01:46.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.396+0000 7f8882ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:46.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.396+0000 7f8882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:46.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.396+0000 7f8882ffd640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:01:46.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.396+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f886c004eb0 con 0x7f888c13f710 2024-03-25T18:01:46.404 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.403+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643470856 0 0) 0x7f888c0ab8b0 con 0x7f888c0a98f0 2024-03-25T18:01:46.404 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.403+0000 7f8882ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:01:46.404 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.403+0000 7f8882ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:46.404 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.403+0000 7f8882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:46.404 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.403+0000 7f8882ffd640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:01:46.404 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.403+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f888c137870 con 0x7f888c0a98f0 2024-03-25T18:01:46.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.405+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888011802 0 0) 0x7f888c0abb90 con 0x7f888c1430c0 2024-03-25T18:01:46.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.405+0000 7f8882ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:01:46.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.405+0000 7f8882ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:46.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.405+0000 7f8882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:46.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.405+0000 7f8882ffd640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:01:46.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.405+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f888c0ab8b0 con 0x7f888c1430c0 2024-03-25T18:01:46.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.405+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2612891547 0 0) 0x7f886c004eb0 con 0x7f888c13f710 2024-03-25T18:01:46.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.405+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f888c0abb90 con 0x7f888c13f710 2024-03-25T18:01:46.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.405+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8874001b00 con 0x7f888c13f710 2024-03-25T18:01:46.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.405+0000 7f8882ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:46.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.405+0000 7f8882ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:01:46.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.405+0000 7f8882ffd640 10 monclient(hunting): dump: 2024-03-25T18:01:46.407 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:46.407 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:46.408 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:46.408 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:46.408 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:46.408 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:46.408 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:46.408 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:46.408 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:46.408 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:46.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.405+0000 7f8882ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:01:46.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3480141773 0 0) 0x7f888c0abb90 con 0x7f888c13f710 2024-03-25T18:01:46.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 >> v1:172.21.15.169:6789/0 conn(0x7f888c1430c0 legacy=0x7f888c1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:46.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 >> v1:172.21.15.18:6789/0 conn(0x7f888c0a98f0 legacy=0x7f888c1350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:46.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8882ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:01:46.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8882ffd640 1 monclient: found mon.c 2024-03-25T18:01:46.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8882ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:01:46.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8882ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:01:46.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888c1389e0 con 0x7f888c13f710 2024-03-25T18:01:46.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8882ffd640 10 monclient: _finish_auth 0 2024-03-25T18:01:46.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8882ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:01:46.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8882ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:01:46.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8890c6a640 5 monclient: authenticate success, global_id 4478 2024-03-25T18:01:46.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8890c6a640 10 monclient: _renew_subs 2024-03-25T18:01:46.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8890c6a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:01:46.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8890c6a640 1 -- 172.21.15.18:0/2855802642 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888c136d90 con 0x7f888c13f710 2024-03-25T18:01:46.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8890c6a640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:01:46.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8890c6a640 10 monclient: _renew_subs 2024-03-25T18:01:46.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8890c6a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:01:46.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8890c6a640 1 -- 172.21.15.18:0/2855802642 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f888c1384c0 con 0x7f888c13f710 2024-03-25T18:01:46.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.418+0000 7f8890c6a640 1 librados: init done 2024-03-25T18:01:46.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.419+0000 7f8890c6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:46.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.419+0000 7f8890c6a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:46.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.419+0000 7f8890c6a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:01:46.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.419+0000 7f8890c6a640 1 -- 172.21.15.18:0/2855802642 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888c136f80 con 0x7f888c13f710 2024-03-25T18:01:46.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.722+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8874001220 con 0x7f888c13f710 2024-03-25T18:01:46.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.722+0000 7f8882ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:01:46.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.722+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8874004c30 con 0x7f888c13f710 2024-03-25T18:01:46.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.722+0000 7f8882ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:46.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.722+0000 7f8882ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:01:46.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.722+0000 7f8882ffd640 10 monclient: dump: 2024-03-25T18:01:46.723 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:46.723 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:46.723 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:46.724 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:46.724 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:46.724 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:46.724 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:46.724 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:46.724 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:46.724 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:46.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.722+0000 7f888959a640 4 set_mon_vals no callback set 2024-03-25T18:01:46.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.723+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f887401ac00 con 0x7f888c13f710 2024-03-25T18:01:46.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.723+0000 7f888a59c640 10 monclient: get_auth_request con 0x7f886c06de00 auth_method 0 2024-03-25T18:01:46.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.723+0000 7f8882ffd640 10 client.4478.objecter ms_handle_connect 0x7f886c06de00 2024-03-25T18:01:46.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.728+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f8874054880 con 0x7f888c13f710 2024-03-25T18:01:46.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.728+0000 7f8882ffd640 10 client.4478.objecter ms_dispatch 0x7f888c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:01:46.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.728+0000 7f8882ffd640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:01:46.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.728+0000 7f8882ffd640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:01:46.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.728+0000 7f8882ffd640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-25T18:01:46.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.728+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f88740501c0 con 0x7f888c13f710 2024-03-25T18:01:46.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.728+0000 7f8882ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:46.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.728+0000 7f8882ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:01:46.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.994+0000 7f8890c6a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:46.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.994+0000 7f8890c6a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:46.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.994+0000 7f8890c6a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:01:46.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:46.994+0000 7f8890c6a640 1 -- 172.21.15.18:0/2855802642 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f888c138b70 con 0x7f888c13f710 2024-03-25T18:01:47.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:47.016+0000 7f8882ffd640 1 -- 172.21.15.18:0/2855802642 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f88740522c0 con 0x7f888c13f710 2024-03-25T18:01:47.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:47.016+0000 7f8882ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:47.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:47.016+0000 7f8882ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:01:47.017 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:01:47.017 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:01:47.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:47.018+0000 7f8890c6a640 10 librados: watch_flush enter 2024-03-25T18:01:47.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:47.018+0000 7f8890c6a640 10 librados: watch_flush exit 2024-03-25T18:01:47.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:47.018+0000 7f8890c6a640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-25T18:01:47.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:47.018+0000 7f8890c6a640 10 client.4478.objecter successfully canceled tick 2024-03-25T18:01:47.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:47.018+0000 7f8890c6a640 1 -- 172.21.15.18:0/2855802642 >> v1:172.21.15.18:6800/3721244451 conn(0x7f886c06de00 legacy=0x7f886c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:47.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:47.018+0000 7f8890c6a640 10 monclient: shutdown 2024-03-25T18:01:47.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:47.018+0000 7f8890c6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:01:47.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:47.018+0000 7f8890c6a640 1 -- 172.21.15.18:0/2855802642 >> v1:172.21.15.18:6790/0 conn(0x7f888c13f710 legacy=0x7f888c1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:47.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:47.018+0000 7f8890c6a640 1 -- 172.21.15.18:0/2855802642 shutdown_connections 2024-03-25T18:01:47.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:47.018+0000 7f8890c6a640 1 -- 172.21.15.18:0/2855802642 >> 172.21.15.18:0/2855802642 conn(0x7f888c0b7ea0 msgr2=0x7f888c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:01:47.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:47.018+0000 7f8890c6a640 1 -- 172.21.15.18:0/2855802642 shutdown_connections 2024-03-25T18:01:47.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:47.018+0000 7f8890c6a640 1 -- 172.21.15.18:0/2855802642 wait complete. 2024-03-25T18:01:47.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:47.018+0000 7f8890c6a640 1 librados: shutdown 2024-03-25T18:01:47.030 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:01:53.837 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:01:53.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.952+0000 7fad7d7fb640 10 monclient: get_monmap_and_config 2024-03-25T18:01:53.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.952+0000 7fad7d7fb640 10 monclient: build_initial_monmap 2024-03-25T18:01:53.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.952+0000 7fad7d7fb640 10 monclient: monmap: 2024-03-25T18:01:53.954 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:01:53.954 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:53.954 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:01:53.953674+0000 2024-03-25T18:01:53.954 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:01:53.953674+0000 2024-03-25T18:01:53.954 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:01:53.954 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:53.954 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:01:53.954 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:01:53.954 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:01:53.955 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:53.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.952+0000 7fad7d7fb640 1 Processor -- start 2024-03-25T18:01:53.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.953+0000 7fad7d7fb640 1 -- start start 2024-03-25T18:01:53.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.953+0000 7fad7d7fb640 10 monclient: init 2024-03-25T18:01:53.956 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.953+0000 7fad7d7fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:01:53.956 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.953+0000 7fad7d7fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:01:53.956 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.953+0000 7fad7d7fb640 10 monclient(hunting): picked mon.noname-a con 0x7fad780a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:01:53.956 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.953+0000 7fad7d7fb640 10 monclient(hunting): picked mon.noname-b con 0x7fad7813f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:01:53.956 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.953+0000 7fad7d7fb640 10 monclient(hunting): picked mon.noname-c con 0x7fad781430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:01:53.956 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.953+0000 7fad7d7fb640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad78142520 con 0x7fad780a98f0 2024-03-25T18:01:53.956 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.953+0000 7fad7d7fb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad780b28a0 con 0x7fad781430c0 2024-03-25T18:01:53.956 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.953+0000 7fad7d7fb640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad780b2a30 con 0x7fad7813f710 2024-03-25T18:01:53.956 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.953+0000 7fad7d7fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:01:53.956 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.953+0000 7fad7d7fb640 10 monclient(hunting): authenticate will time out at 7242.532258s 2024-03-25T18:01:53.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.966+0000 7fad777fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fad781430c0 0x7fad7814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:41808/0 (socket says 172.21.15.18:41808) 2024-03-25T18:01:53.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.966+0000 7fad777fe640 1 -- 172.21.15.18:0/1458311460 learned_addr learned my addr 172.21.15.18:0/1458311460 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:01:53.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.975+0000 7fad75ffb640 1 -- 172.21.15.18:0/1458311460 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215446796 0 0) 0x7fad780b2a30 con 0x7fad7813f710 2024-03-25T18:01:53.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.975+0000 7fad75ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:01:53.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.975+0000 7fad75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:53.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.975+0000 7fad75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:53.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.975+0000 7fad75ffb640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:01:53.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.975+0000 7fad75ffb640 1 -- 172.21.15.18:0/1458311460 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad60003520 con 0x7fad7813f710 2024-03-25T18:01:53.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.989+0000 7fad75ffb640 1 -- 172.21.15.18:0/1458311460 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287345086 0 0) 0x7fad780b28a0 con 0x7fad781430c0 2024-03-25T18:01:53.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.989+0000 7fad75ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:01:53.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.989+0000 7fad75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:53.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.989+0000 7fad75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:53.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.989+0000 7fad75ffb640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:01:53.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.989+0000 7fad75ffb640 1 -- 172.21.15.18:0/1458311460 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad780b2a30 con 0x7fad781430c0 2024-03-25T18:01:53.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.993+0000 7fad75ffb640 1 -- 172.21.15.18:0/1458311460 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339804188 0 0) 0x7fad78142520 con 0x7fad780a98f0 2024-03-25T18:01:53.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.993+0000 7fad75ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:01:53.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.993+0000 7fad75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:53.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.993+0000 7fad75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:53.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.993+0000 7fad75ffb640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:01:53.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.993+0000 7fad75ffb640 1 -- 172.21.15.18:0/1458311460 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad780b28a0 con 0x7fad780a98f0 2024-03-25T18:01:54.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:53.999+0000 7fad75ffb640 1 -- 172.21.15.18:0/1458311460 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 292294184 0 0) 0x7fad60003520 con 0x7fad7813f710 2024-03-25T18:01:54.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.000+0000 7fad75ffb640 1 -- 172.21.15.18:0/1458311460 >> v1:172.21.15.169:6789/0 conn(0x7fad781430c0 legacy=0x7fad7814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:54.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.000+0000 7fad75ffb640 1 -- 172.21.15.18:0/1458311460 >> v1:172.21.15.18:6789/0 conn(0x7fad780a98f0 legacy=0x7fad780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:54.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.000+0000 7fad75ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:01:54.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.000+0000 7fad75ffb640 1 monclient: found mon.noname-b 2024-03-25T18:01:54.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.000+0000 7fad75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:01:54.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.000+0000 7fad75ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:01:54.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.000+0000 7fad75ffb640 1 -- 172.21.15.18:0/1458311460 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad780b44c0 con 0x7fad7813f710 2024-03-25T18:01:54.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.000+0000 7fad75ffb640 10 monclient: _finish_auth 0 2024-03-25T18:01:54.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.000+0000 7fad75ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:01:54.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.000+0000 7fad75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:01:54.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.000+0000 7fad75ffb640 1 -- 172.21.15.18:0/1458311460 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fad6c001690 con 0x7fad7813f710 2024-03-25T18:01:54.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.000+0000 7fad7d7fb640 5 monclient: authenticate success, global_id 4481 2024-03-25T18:01:54.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.000+0000 7fad7d7fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:01:54.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.000+0000 7fad75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:54.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.000+0000 7fad75ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:01:54.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.000+0000 7fad75ffb640 10 monclient: dump: 2024-03-25T18:01:54.003 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:54.003 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:54.003 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:54.003 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:54.003 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:54.003 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:54.003 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:54.003 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:54.003 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:54.003 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:54.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.000+0000 7fad7d7fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:01:54.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.012+0000 7fad75ffb640 1 -- 172.21.15.18:0/1458311460 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad6c0017f0 con 0x7fad7813f710 2024-03-25T18:01:54.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.012+0000 7fad75ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:01:54.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.012+0000 7fad75ffb640 1 -- 172.21.15.18:0/1458311460 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fad6c001c00 con 0x7fad7813f710 2024-03-25T18:01:54.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.012+0000 7fad75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:54.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.012+0000 7fad75ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:01:54.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.012+0000 7fad75ffb640 10 monclient: dump: 2024-03-25T18:01:54.013 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:54.014 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:54.014 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:54.014 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:54.014 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:54.014 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:54.014 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:54.014 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:54.014 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:54.014 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:54.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.012+0000 7fad7d7fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:01:54.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.012+0000 7fad7d7fb640 4 set_mon_vals no callback set 2024-03-25T18:01:54.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.012+0000 7fad7d7fb640 10 monclient: shutdown 2024-03-25T18:01:54.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.012+0000 7fad7d7fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:01:54.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.012+0000 7fad7d7fb640 1 -- 172.21.15.18:0/1458311460 >> v1:172.21.15.18:6790/0 conn(0x7fad7813f710 legacy=0x7fad781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:54.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.013+0000 7fad7d7fb640 1 -- 172.21.15.18:0/1458311460 shutdown_connections 2024-03-25T18:01:54.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.013+0000 7fad7d7fb640 1 -- 172.21.15.18:0/1458311460 >> 172.21.15.18:0/1458311460 conn(0x7fad780b7ea0 msgr2=0x7fad780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:01:54.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.013+0000 7fad7d7fb640 1 -- 172.21.15.18:0/1458311460 shutdown_connections 2024-03-25T18:01:54.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.013+0000 7fad7d7fb640 1 -- 172.21.15.18:0/1458311460 wait complete. 2024-03-25T18:01:54.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.014+0000 7fad7d7fb640 10 monclient: build_initial_monmap 2024-03-25T18:01:54.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.014+0000 7fad7d7fb640 10 monclient: monmap: 2024-03-25T18:01:54.018 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:01:54.018 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:01:54.018 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:01:54.018 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:01:54.018 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:01:54.019 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:01:54.019 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:01:54.019 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:01:54.019 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:01:54.019 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:54.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.014+0000 7fad7d7fb640 1 librados: starting msgr at 2024-03-25T18:01:54.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.014+0000 7fad7d7fb640 1 librados: starting objecter 2024-03-25T18:01:54.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.014+0000 7fad7d7fb640 1 Processor -- start 2024-03-25T18:01:54.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.014+0000 7fad7d7fb640 1 -- start start 2024-03-25T18:01:54.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.014+0000 7fad7d7fb640 1 librados: setting wanted keys 2024-03-25T18:01:54.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.014+0000 7fad7d7fb640 1 librados: calling monclient init 2024-03-25T18:01:54.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.014+0000 7fad7d7fb640 10 monclient: init 2024-03-25T18:01:54.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.015+0000 7fad7d7fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:01:54.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.015+0000 7fad7d7fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:01:54.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.015+0000 7fad7d7fb640 10 monclient(hunting): picked mon.noname-c con 0x7fad780a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:01:54.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.015+0000 7fad7d7fb640 10 monclient(hunting): picked mon.noname-a con 0x7fad7813f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:01:54.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.015+0000 7fad7d7fb640 10 monclient(hunting): picked mon.noname-b con 0x7fad781430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:01:54.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.015+0000 7fad7d7fb640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad781db9a0 con 0x7fad7813f710 2024-03-25T18:01:54.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.015+0000 7fad7d7fb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad781dc5d0 con 0x7fad781430c0 2024-03-25T18:01:54.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.015+0000 7fad7d7fb640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad781dc760 con 0x7fad780a98f0 2024-03-25T18:01:54.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.015+0000 7fad7d7fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:01:54.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.015+0000 7fad7d7fb640 10 monclient(hunting): authenticate will time out at 7242.594628s 2024-03-25T18:01:54.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.024+0000 7fad777fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fad781430c0 0x7fad781e6360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:41882/0 (socket says 172.21.15.18:41882) 2024-03-25T18:01:54.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.024+0000 7fad777fe640 1 -- 172.21.15.18:0/76884183 learned_addr learned my addr 172.21.15.18:0/76884183 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:01:54.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.024+0000 7fad5f7fe640 10 client.?.objecter ms_handle_connect 0x7fad781430c0 2024-03-25T18:01:54.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.025+0000 7fad5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:54.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.029+0000 7fad5f7fe640 10 client.?.objecter ms_handle_connect 0x7fad780a98f0 2024-03-25T18:01:54.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.029+0000 7fad5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:54.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.034+0000 7fad5f7fe640 10 client.?.objecter ms_handle_connect 0x7fad7813f710 2024-03-25T18:01:54.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.034+0000 7fad5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:01:54.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.035+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753371742 0 0) 0x7fad781dc760 con 0x7fad780a98f0 2024-03-25T18:01:54.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.035+0000 7fad5f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:01:54.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.035+0000 7fad5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:54.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.035+0000 7fad5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:54.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.035+0000 7fad5f7fe640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:01:54.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.035+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad64004eb0 con 0x7fad780a98f0 2024-03-25T18:01:54.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.038+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950489099 0 0) 0x7fad781dc5d0 con 0x7fad781430c0 2024-03-25T18:01:54.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.038+0000 7fad5f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:01:54.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.038+0000 7fad5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:54.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.038+0000 7fad5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:54.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.038+0000 7fad5f7fe640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:01:54.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.038+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad781dc760 con 0x7fad781430c0 2024-03-25T18:01:54.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.038+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517754646 0 0) 0x7fad781db9a0 con 0x7fad7813f710 2024-03-25T18:01:54.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.038+0000 7fad5f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:01:54.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.038+0000 7fad5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:01:54.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.038+0000 7fad5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:01:54.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.038+0000 7fad5f7fe640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:01:54.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.038+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad781dc5d0 con 0x7fad7813f710 2024-03-25T18:01:54.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.049+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4159690622 0 0) 0x7fad64004eb0 con 0x7fad780a98f0 2024-03-25T18:01:54.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.049+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad781db9a0 con 0x7fad780a98f0 2024-03-25T18:01:54.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.049+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fad680018f0 con 0x7fad780a98f0 2024-03-25T18:01:54.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.049+0000 7fad5f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:54.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.049+0000 7fad5f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:01:54.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.049+0000 7fad5f7fe640 10 monclient(hunting): dump: 2024-03-25T18:01:54.052 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:54.052 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:54.052 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:54.052 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:54.052 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:54.052 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:54.052 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:54.052 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:54.052 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:54.052 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:54.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.049+0000 7fad5f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:01:54.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.050+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2257722572 0 0) 0x7fad781dc5d0 con 0x7fad7813f710 2024-03-25T18:01:54.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.050+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad64004eb0 con 0x7fad7813f710 2024-03-25T18:01:54.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.050+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fad6c001b00 con 0x7fad7813f710 2024-03-25T18:01:54.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.050+0000 7fad5f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:54.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.050+0000 7fad5f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:01:54.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.050+0000 7fad5f7fe640 10 monclient(hunting): dump: 2024-03-25T18:01:54.053 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:54.053 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:54.053 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:54.053 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:54.053 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:54.053 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:54.053 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:54.053 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:54.054 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:54.054 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:54.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.051+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 963417048 0 0) 0x7fad781dc760 con 0x7fad781430c0 2024-03-25T18:01:54.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.051+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad781dc5d0 con 0x7fad781430c0 2024-03-25T18:01:54.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.051+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fad70001bd0 con 0x7fad781430c0 2024-03-25T18:01:54.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.051+0000 7fad5f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:54.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.051+0000 7fad5f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:01:54.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.051+0000 7fad5f7fe640 10 monclient(hunting): dump: 2024-03-25T18:01:54.055 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:54.055 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:54.055 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:54.055 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:54.055 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:54.055 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:54.055 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:54.055 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:54.055 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:54.056 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:54.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.056+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2790526338 0 0) 0x7fad64004eb0 con 0x7fad7813f710 2024-03-25T18:01:54.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.056+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 >> v1:172.21.15.18:6790/0 conn(0x7fad780a98f0 legacy=0x7fad781dbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:54.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.056+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 >> v1:172.21.15.169:6789/0 conn(0x7fad781430c0 legacy=0x7fad781e6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:54.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.056+0000 7fad5f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:01:54.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.056+0000 7fad5f7fe640 1 monclient: found mon.a 2024-03-25T18:01:54.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.056+0000 7fad5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:01:54.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.056+0000 7fad5f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:01:54.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.056+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7806dea0 con 0x7fad7813f710 2024-03-25T18:01:54.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.056+0000 7fad5f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:01:54.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.056+0000 7fad5f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:01:54.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.056+0000 7fad5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:01:54.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.056+0000 7fad7d7fb640 5 monclient: authenticate success, global_id 4476 2024-03-25T18:01:54.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.056+0000 7fad7d7fb640 10 monclient: _renew_subs 2024-03-25T18:01:54.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.056+0000 7fad7d7fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:01:54.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.056+0000 7fad7d7fb640 1 -- 172.21.15.18:0/76884183 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7806f1a0 con 0x7fad7813f710 2024-03-25T18:01:54.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.057+0000 7fad7d7fb640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:01:54.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.057+0000 7fad7d7fb640 10 monclient: _renew_subs 2024-03-25T18:01:54.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.057+0000 7fad7d7fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:01:54.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.057+0000 7fad7d7fb640 1 -- 172.21.15.18:0/76884183 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad78070b30 con 0x7fad7813f710 2024-03-25T18:01:54.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.057+0000 7fad7d7fb640 1 librados: init done 2024-03-25T18:01:54.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.057+0000 7fad7d7fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:54.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.057+0000 7fad7d7fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:54.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.057+0000 7fad7d7fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:01:54.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.057+0000 7fad7d7fb640 1 -- 172.21.15.18:0/76884183 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad7806e0f0 con 0x7fad7813f710 2024-03-25T18:01:54.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.081+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad6c001c60 con 0x7fad7813f710 2024-03-25T18:01:54.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.081+0000 7fad5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:01:54.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.081+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fad6c003a40 con 0x7fad7813f710 2024-03-25T18:01:54.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.081+0000 7fad5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:01:54.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.081+0000 7fad5f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:01:54.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.081+0000 7fad5f7fe640 10 monclient: dump: 2024-03-25T18:01:54.084 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:01:54.084 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:01:54.084 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:54.084 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:01:54.085 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:01:54.085 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:01:54.085 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:01:54.085 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:01:54.085 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:01:54.085 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:01:54.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.081+0000 7fad757fa640 4 set_mon_vals no callback set 2024-03-25T18:01:54.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.081+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fad6c019a10 con 0x7fad7813f710 2024-03-25T18:01:54.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.082+0000 7fad76ffd640 10 monclient: get_auth_request con 0x7fad6406e100 auth_method 0 2024-03-25T18:01:54.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.082+0000 7fad5f7fe640 10 client.4476.objecter ms_handle_connect 0x7fad6406e100 2024-03-25T18:01:54.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.085+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fad6c0535e0 con 0x7fad7813f710 2024-03-25T18:01:54.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.085+0000 7fad5f7fe640 10 client.4476.objecter ms_dispatch 0x7fad78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:01:54.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.085+0000 7fad5f7fe640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:01:54.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.085+0000 7fad5f7fe640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:01:54.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.085+0000 7fad5f7fe640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-25T18:01:54.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.396+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fad6c04f0f0 con 0x7fad7813f710 2024-03-25T18:01:54.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.396+0000 7fad5f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:01:54.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.396+0000 7fad5f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:01:54.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.662+0000 7fad7d7fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:54.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.662+0000 7fad7d7fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:54.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.662+0000 7fad7d7fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:01:54.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.662+0000 7fad7d7fb640 1 -- 172.21.15.18:0/76884183 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad78070630 con 0x7fad7813f710 2024-03-25T18:01:54.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.705+0000 7fad5f7fe640 1 -- 172.21.15.18:0/76884183 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fad6c051020 con 0x7fad7813f710 2024-03-25T18:01:54.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.705+0000 7fad5f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:01:54.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.705+0000 7fad5f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:01:54.707 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:01:54.707 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:01:54.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.707+0000 7fad7d7fb640 10 librados: watch_flush enter 2024-03-25T18:01:54.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.707+0000 7fad7d7fb640 10 librados: watch_flush exit 2024-03-25T18:01:54.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.707+0000 7fad7d7fb640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-25T18:01:54.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.707+0000 7fad7d7fb640 10 client.4476.objecter successfully canceled tick 2024-03-25T18:01:54.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.707+0000 7fad7d7fb640 1 -- 172.21.15.18:0/76884183 >> v1:172.21.15.18:6800/3721244451 conn(0x7fad6406e100 legacy=0x7fad640705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:54.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.707+0000 7fad7d7fb640 10 monclient: shutdown 2024-03-25T18:01:54.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.707+0000 7fad7d7fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:01:54.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.707+0000 7fad7d7fb640 1 -- 172.21.15.18:0/76884183 >> v1:172.21.15.18:6789/0 conn(0x7fad7813f710 legacy=0x7fad7806ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:01:54.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.708+0000 7fad7d7fb640 1 -- 172.21.15.18:0/76884183 shutdown_connections 2024-03-25T18:01:54.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.708+0000 7fad7d7fb640 1 -- 172.21.15.18:0/76884183 >> 172.21.15.18:0/76884183 conn(0x7fad780b7ea0 msgr2=0x7fad78074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:01:54.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.708+0000 7fad7d7fb640 1 -- 172.21.15.18:0/76884183 shutdown_connections 2024-03-25T18:01:54.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.708+0000 7fad7d7fb640 1 -- 172.21.15.18:0/76884183 wait complete. 2024-03-25T18:01:54.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:01:54.708+0000 7fad7d7fb640 1 librados: shutdown 2024-03-25T18:01:54.720 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:02:01.526 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:02:01.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.641+0000 7fd53ab4b640 10 monclient: get_monmap_and_config 2024-03-25T18:02:01.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.641+0000 7fd53ab4b640 10 monclient: build_initial_monmap 2024-03-25T18:02:01.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.641+0000 7fd53ab4b640 10 monclient: monmap: 2024-03-25T18:02:01.653 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:02:01.653 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:01.653 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:02:01.642610+0000 2024-03-25T18:02:01.653 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:02:01.642610+0000 2024-03-25T18:02:01.653 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:02:01.653 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:01.653 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:02:01.653 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:02:01.653 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:02:01.653 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:01.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.642+0000 7fd53ab4b640 1 Processor -- start 2024-03-25T18:02:01.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.642+0000 7fd53ab4b640 1 -- start start 2024-03-25T18:02:01.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.642+0000 7fd53ab4b640 10 monclient: init 2024-03-25T18:02:01.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.643+0000 7fd53ab4b640 10 monclient: _reopen_session rank -1 2024-03-25T18:02:01.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.643+0000 7fd53ab4b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:02:01.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.643+0000 7fd53ab4b640 10 monclient(hunting): picked mon.noname-a con 0x7fd5340b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T18:02:01.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.643+0000 7fd53ab4b640 10 monclient(hunting): picked mon.noname-c con 0x7fd5340b1730 addr v1:172.21.15.169:6789/0 2024-03-25T18:02:01.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.643+0000 7fd53ab4b640 10 monclient(hunting): picked mon.noname-b con 0x7fd53414be90 addr v1:172.21.15.18:6790/0 2024-03-25T18:02:01.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.643+0000 7fd53ab4b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5340af940 con 0x7fd5340b4d60 2024-03-25T18:02:01.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.643+0000 7fd53ab4b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5340b0a70 con 0x7fd5340b1730 2024-03-25T18:02:01.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.643+0000 7fd53ab4b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5340b0d50 con 0x7fd53414be90 2024-03-25T18:02:01.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.643+0000 7fd53ab4b640 10 monclient(hunting): _renew_subs 2024-03-25T18:02:01.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.643+0000 7fd53ab4b640 10 monclient(hunting): authenticate will time out at 7250.222110s 2024-03-25T18:02:01.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.649+0000 7fd533fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fd5340b1730 0x7fd5340b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:48084/0 (socket says 172.21.15.18:48084) 2024-03-25T18:02:01.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.649+0000 7fd533fff640 1 -- 172.21.15.18:0/736828140 learned_addr learned my addr 172.21.15.18:0/736828140 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:02:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.664+0000 7fd5337fe640 1 -- 172.21.15.18:0/736828140 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144618780 0 0) 0x7fd5340b0d50 con 0x7fd53414be90 2024-03-25T18:02:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.664+0000 7fd5337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:02:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.664+0000 7fd5337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.664+0000 7fd5337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.664+0000 7fd5337fe640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:02:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.664+0000 7fd5337fe640 1 -- 172.21.15.18:0/736828140 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd524003520 con 0x7fd53414be90 2024-03-25T18:02:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.665+0000 7fd5337fe640 1 -- 172.21.15.18:0/736828140 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603353943 0 0) 0x7fd5340b0a70 con 0x7fd5340b1730 2024-03-25T18:02:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.665+0000 7fd5337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:02:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.665+0000 7fd5337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.665+0000 7fd5337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.665+0000 7fd5337fe640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:02:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.665+0000 7fd5337fe640 1 -- 172.21.15.18:0/736828140 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5340b0d50 con 0x7fd5340b1730 2024-03-25T18:02:01.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.674+0000 7fd5337fe640 1 -- 172.21.15.18:0/736828140 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3845077941 0 0) 0x7fd524003520 con 0x7fd53414be90 2024-03-25T18:02:01.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.674+0000 7fd5337fe640 1 -- 172.21.15.18:0/736828140 >> v1:172.21.15.169:6789/0 conn(0x7fd5340b1730 legacy=0x7fd5340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:01.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.674+0000 7fd5337fe640 1 -- 172.21.15.18:0/736828140 >> v1:172.21.15.18:6789/0 conn(0x7fd5340b4d60 legacy=0x7fd53413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:01.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.674+0000 7fd5337fe640 10 monclient: _finish_hunting 0 2024-03-25T18:02:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.674+0000 7fd5337fe640 1 monclient: found mon.noname-b 2024-03-25T18:02:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.674+0000 7fd5337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:02:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.674+0000 7fd5337fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:02:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.674+0000 7fd5337fe640 1 -- 172.21.15.18:0/736828140 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5340b1030 con 0x7fd53414be90 2024-03-25T18:02:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.674+0000 7fd5337fe640 10 monclient: _finish_auth 0 2024-03-25T18:02:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.674+0000 7fd5337fe640 10 monclient: _check_auth_tickets 2024-03-25T18:02:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.674+0000 7fd5337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:02:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.674+0000 7fd5337fe640 1 -- 172.21.15.18:0/736828140 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd52c001a10 con 0x7fd53414be90 2024-03-25T18:02:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.674+0000 7fd5337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.674+0000 7fd5337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:02:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.674+0000 7fd5337fe640 10 monclient: dump: 2024-03-25T18:02:01.676 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:01.676 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:01.676 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:01.677 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:01.677 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:01.677 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:01.677 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:01.677 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:01.677 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:01.677 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:01.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.674+0000 7fd53ab4b640 5 monclient: authenticate success, global_id 4487 2024-03-25T18:02:01.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.674+0000 7fd53ab4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:02:01.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.688+0000 7fd5337fe640 1 -- 172.21.15.18:0/736828140 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd52c001b70 con 0x7fd53414be90 2024-03-25T18:02:01.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.688+0000 7fd5337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:02:01.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.688+0000 7fd5337fe640 1 -- 172.21.15.18:0/736828140 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd52c001f80 con 0x7fd53414be90 2024-03-25T18:02:01.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.688+0000 7fd5337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:01.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.688+0000 7fd5337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:02:01.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.688+0000 7fd5337fe640 10 monclient: dump: 2024-03-25T18:02:01.689 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:01.689 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:01.689 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:01.689 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:01.689 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:01.690 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:01.690 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:01.690 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:01.690 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:01.690 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:01.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.688+0000 7fd53ab4b640 10 monclient: get_monmap_and_config success 2024-03-25T18:02:01.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.688+0000 7fd53ab4b640 4 set_mon_vals no callback set 2024-03-25T18:02:01.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.689+0000 7fd53ab4b640 10 monclient: shutdown 2024-03-25T18:02:01.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.689+0000 7fd53ab4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:02:01.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.689+0000 7fd53ab4b640 1 -- 172.21.15.18:0/736828140 >> v1:172.21.15.18:6790/0 conn(0x7fd53414be90 legacy=0x7fd53414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:01.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.689+0000 7fd53ab4b640 1 -- 172.21.15.18:0/736828140 shutdown_connections 2024-03-25T18:02:01.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.689+0000 7fd53ab4b640 1 -- 172.21.15.18:0/736828140 >> 172.21.15.18:0/736828140 conn(0x7fd5340617a0 msgr2=0x7fd534061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:02:01.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.689+0000 7fd53ab4b640 1 -- 172.21.15.18:0/736828140 shutdown_connections 2024-03-25T18:02:01.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.689+0000 7fd53ab4b640 1 -- 172.21.15.18:0/736828140 wait complete. 2024-03-25T18:02:01.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.690+0000 7fd53ab4b640 10 monclient: build_initial_monmap 2024-03-25T18:02:01.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.690+0000 7fd53ab4b640 10 monclient: monmap: 2024-03-25T18:02:01.692 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:02:01.692 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:02:01.692 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:02:01.692 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:02:01.693 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:02:01.693 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:02:01.693 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:02:01.693 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:02:01.693 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:02:01.693 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:01.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.690+0000 7fd53ab4b640 1 librados: starting msgr at 2024-03-25T18:02:01.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.690+0000 7fd53ab4b640 1 librados: starting objecter 2024-03-25T18:02:01.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.690+0000 7fd53ab4b640 1 Processor -- start 2024-03-25T18:02:01.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.691+0000 7fd53ab4b640 1 -- start start 2024-03-25T18:02:01.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.691+0000 7fd53ab4b640 1 librados: setting wanted keys 2024-03-25T18:02:01.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.691+0000 7fd53ab4b640 1 librados: calling monclient init 2024-03-25T18:02:01.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.691+0000 7fd53ab4b640 10 monclient: init 2024-03-25T18:02:01.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.692+0000 7fd53ab4b640 10 monclient: _reopen_session rank -1 2024-03-25T18:02:01.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.692+0000 7fd53ab4b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:02:01.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.692+0000 7fd53ab4b640 10 monclient(hunting): picked mon.noname-c con 0x7fd5340b1730 addr v1:172.21.15.18:6790/0 2024-03-25T18:02:01.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.692+0000 7fd53ab4b640 10 monclient(hunting): picked mon.noname-a con 0x7fd5340b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T18:02:01.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.692+0000 7fd53ab4b640 10 monclient(hunting): picked mon.noname-b con 0x7fd53414be90 addr v1:172.21.15.169:6789/0 2024-03-25T18:02:01.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.692+0000 7fd53ab4b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd534141800 con 0x7fd5340b4d60 2024-03-25T18:02:01.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.692+0000 7fd53ab4b640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd534143490 con 0x7fd53414be90 2024-03-25T18:02:01.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.692+0000 7fd53ab4b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd534143620 con 0x7fd5340b1730 2024-03-25T18:02:01.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.692+0000 7fd53ab4b640 10 monclient(hunting): _renew_subs 2024-03-25T18:02:01.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.692+0000 7fd53ab4b640 10 monclient(hunting): authenticate will time out at 7250.270923s 2024-03-25T18:02:01.698 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.697+0000 7fd533fff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fd5340b4d60 0x7fd534142cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:51202/0 (socket says 172.21.15.18:51202) 2024-03-25T18:02:01.698 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.697+0000 7fd533fff640 1 -- 172.21.15.18:0/2945191644 learned_addr learned my addr 172.21.15.18:0/2945191644 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:02:01.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.699+0000 7fd5317fa640 10 client.?.objecter ms_handle_connect 0x7fd5340b4d60 2024-03-25T18:02:01.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.699+0000 7fd5317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:01.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.706+0000 7fd5317fa640 10 client.?.objecter ms_handle_connect 0x7fd53414be90 2024-03-25T18:02:01.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.706+0000 7fd5317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:01.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.711+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671674653 0 0) 0x7fd534141800 con 0x7fd5340b4d60 2024-03-25T18:02:01.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.711+0000 7fd5317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:02:01.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.711+0000 7fd5317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:01.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.711+0000 7fd5317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:01.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.711+0000 7fd5317fa640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:02:01.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.711+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd518004eb0 con 0x7fd5340b4d60 2024-03-25T18:02:01.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.712+0000 7fd5317fa640 10 client.?.objecter ms_handle_connect 0x7fd5340b1730 2024-03-25T18:02:01.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.712+0000 7fd5317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:01.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.715+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260785561 0 0) 0x7fd534143490 con 0x7fd53414be90 2024-03-25T18:02:01.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.715+0000 7fd5317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:02:01.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.715+0000 7fd5317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:01.718 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.715+0000 7fd5317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:01.718 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.715+0000 7fd5317fa640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:02:01.718 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.715+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd534141800 con 0x7fd53414be90 2024-03-25T18:02:01.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.724+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694603295 0 0) 0x7fd534143620 con 0x7fd5340b1730 2024-03-25T18:02:01.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.724+0000 7fd5317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:02:01.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.724+0000 7fd5317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:01.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.724+0000 7fd5317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:01.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.724+0000 7fd5317fa640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:02:01.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.724+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd534143490 con 0x7fd5340b1730 2024-03-25T18:02:01.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.747+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3929673629 0 0) 0x7fd534141800 con 0x7fd53414be90 2024-03-25T18:02:01.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.747+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd534143620 con 0x7fd53414be90 2024-03-25T18:02:01.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.747+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd52c001e80 con 0x7fd53414be90 2024-03-25T18:02:01.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.747+0000 7fd5317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:01.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.747+0000 7fd5317fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:02:01.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.747+0000 7fd5317fa640 10 monclient(hunting): dump: 2024-03-25T18:02:01.749 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:01.749 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:01.749 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:01.750 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:01.750 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:01.750 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:01.750 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:01.750 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:01.750 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:01.750 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:01.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.747+0000 7fd5317fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:02:01.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.748+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 553175778 0 0) 0x7fd518004eb0 con 0x7fd5340b4d60 2024-03-25T18:02:01.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.748+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd534141800 con 0x7fd5340b4d60 2024-03-25T18:02:01.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.748+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd5200019f0 con 0x7fd5340b4d60 2024-03-25T18:02:01.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.748+0000 7fd5317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:01.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.748+0000 7fd5317fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:02:01.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.748+0000 7fd5317fa640 10 monclient(hunting): dump: 2024-03-25T18:02:01.751 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:01.751 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:01.751 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:01.751 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:01.751 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:01.751 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:01.751 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:01.751 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:01.751 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:01.751 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:01.757 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.756+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3027262302 0 0) 0x7fd534143490 con 0x7fd5340b1730 2024-03-25T18:02:01.757 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.756+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd518004eb0 con 0x7fd5340b1730 2024-03-25T18:02:01.758 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.756+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd528001870 con 0x7fd5340b1730 2024-03-25T18:02:01.758 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.756+0000 7fd5317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:01.758 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.756+0000 7fd5317fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:02:01.758 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.756+0000 7fd5317fa640 10 monclient(hunting): dump: 2024-03-25T18:02:01.758 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:01.758 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:01.758 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:01.758 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:01.758 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:01.758 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:01.758 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:01.758 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:01.759 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:01.759 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:01.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3783175086 0 0) 0x7fd534141800 con 0x7fd5340b4d60 2024-03-25T18:02:01.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 >> v1:172.21.15.18:6790/0 conn(0x7fd5340b1730 legacy=0x7fd5341425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:01.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 >> v1:172.21.15.169:6789/0 conn(0x7fd53414be90 legacy=0x7fd5341e6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:01.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd5317fa640 10 monclient: _finish_hunting 0 2024-03-25T18:02:01.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd5317fa640 1 monclient: found mon.a 2024-03-25T18:02:01.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd5317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:02:01.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd5317fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:02:01.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53413bd50 con 0x7fd5340b4d60 2024-03-25T18:02:01.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd5317fa640 10 monclient: _finish_auth 0 2024-03-25T18:02:01.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd5317fa640 10 monclient: _check_auth_tickets 2024-03-25T18:02:01.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd5317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:02:01.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd53ab4b640 5 monclient: authenticate success, global_id 4482 2024-03-25T18:02:01.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd53ab4b640 10 monclient: _renew_subs 2024-03-25T18:02:01.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd53ab4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:02:01.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd53ab4b640 1 -- 172.21.15.18:0/2945191644 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53413bfa0 con 0x7fd5340b4d60 2024-03-25T18:02:01.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd53ab4b640 10 client.4482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:02:01.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd53ab4b640 10 monclient: _renew_subs 2024-03-25T18:02:01.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd53ab4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:02:01.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd53ab4b640 1 -- 172.21.15.18:0/2945191644 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5341407d0 con 0x7fd5340b4d60 2024-03-25T18:02:01.767 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.764+0000 7fd53ab4b640 1 librados: init done 2024-03-25T18:02:01.768 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.765+0000 7fd53ab4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:01.768 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.765+0000 7fd53ab4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:01.768 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.765+0000 7fd53ab4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:02:01.768 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:01.765+0000 7fd53ab4b640 1 -- 172.21.15.18:0/2945191644 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd534142410 con 0x7fd5340b4d60 2024-03-25T18:02:02.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.087+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5200047e0 con 0x7fd5340b4d60 2024-03-25T18:02:02.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.087+0000 7fd5317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:02:02.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.087+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd520004af0 con 0x7fd5340b4d60 2024-03-25T18:02:02.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.087+0000 7fd5317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:02.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.087+0000 7fd5317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:02:02.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.087+0000 7fd5317fa640 10 monclient: dump: 2024-03-25T18:02:02.089 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:02.090 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:02.090 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:02.090 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:02.090 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:02.090 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:02.090 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:02.090 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:02.090 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:02.090 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:02.090 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.087+0000 7fd532ffd640 4 set_mon_vals no callback set 2024-03-25T18:02:02.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.088+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fd52001ab40 con 0x7fd5340b4d60 2024-03-25T18:02:02.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.089+0000 7fd5388e7640 10 monclient: get_auth_request con 0x7fd51806e110 auth_method 0 2024-03-25T18:02:02.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.089+0000 7fd5317fa640 10 client.4482.objecter ms_handle_connect 0x7fd51806e110 2024-03-25T18:02:02.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.094+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fd520054880 con 0x7fd5340b4d60 2024-03-25T18:02:02.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.094+0000 7fd5317fa640 10 client.4482.objecter ms_dispatch 0x7fd534000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:02:02.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.095+0000 7fd5317fa640 3 client.4482.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:02:02.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.095+0000 7fd5317fa640 3 client.4482.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:02:02.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.095+0000 7fd5317fa640 20 client.4482.objecter dump_active .. 0 homeless 2024-03-25T18:02:02.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.095+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd5200501c0 con 0x7fd5340b4d60 2024-03-25T18:02:02.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.095+0000 7fd5317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:02.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.095+0000 7fd5317fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:02:02.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.318+0000 7fd53ab4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:02.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.318+0000 7fd53ab4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:02.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.318+0000 7fd53ab4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:02:02.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.318+0000 7fd53ab4b640 1 -- 172.21.15.18:0/2945191644 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd534141f00 con 0x7fd5340b4d60 2024-03-25T18:02:02.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.369+0000 7fd5317fa640 1 -- 172.21.15.18:0/2945191644 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fd534141f00 con 0x7fd5340b4d60 2024-03-25T18:02:02.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.369+0000 7fd5317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:02.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.369+0000 7fd5317fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:02:02.370 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:02:02.370 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:02:02.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.371+0000 7fd53ab4b640 10 librados: watch_flush enter 2024-03-25T18:02:02.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.371+0000 7fd53ab4b640 10 librados: watch_flush exit 2024-03-25T18:02:02.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.371+0000 7fd53ab4b640 20 client.4482.objecter shutdown clearing up homeless session... 2024-03-25T18:02:02.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.371+0000 7fd53ab4b640 10 client.4482.objecter successfully canceled tick 2024-03-25T18:02:02.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.371+0000 7fd53ab4b640 1 -- 172.21.15.18:0/2945191644 >> v1:172.21.15.18:6800/3721244451 conn(0x7fd51806e110 legacy=0x7fd5180705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:02.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.371+0000 7fd53ab4b640 10 monclient: shutdown 2024-03-25T18:02:02.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.371+0000 7fd53ab4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:02:02.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.371+0000 7fd53ab4b640 1 -- 172.21.15.18:0/2945191644 >> v1:172.21.15.18:6789/0 conn(0x7fd5340b4d60 legacy=0x7fd534142cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:02.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.371+0000 7fd53ab4b640 1 -- 172.21.15.18:0/2945191644 shutdown_connections 2024-03-25T18:02:02.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.371+0000 7fd53ab4b640 1 -- 172.21.15.18:0/2945191644 >> 172.21.15.18:0/2945191644 conn(0x7fd5340617a0 msgr2=0x7fd5340b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:02:02.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.372+0000 7fd53ab4b640 1 -- 172.21.15.18:0/2945191644 shutdown_connections 2024-03-25T18:02:02.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.372+0000 7fd53ab4b640 1 -- 172.21.15.18:0/2945191644 wait complete. 2024-03-25T18:02:02.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:02.372+0000 7fd53ab4b640 1 librados: shutdown 2024-03-25T18:02:02.384 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:02:03.022 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:06:00.814 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:03.028 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:06:00.821 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:03.071 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:00.820 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:03.073 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:00.822 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:03.074 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:06:00.867 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:03.076 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:06:00.869 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:03.087 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:00.836 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:03.091 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:06:00.884 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:03.121 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:06:00.914 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:09.191 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:02:09.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.306+0000 7f3ad044e640 10 monclient: get_monmap_and_config 2024-03-25T18:02:09.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.306+0000 7f3ad044e640 10 monclient: build_initial_monmap 2024-03-25T18:02:09.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.306+0000 7f3ad044e640 10 monclient: monmap: 2024-03-25T18:02:09.308 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:02:09.308 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:09.308 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:02:09.307440+0000 2024-03-25T18:02:09.308 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:02:09.307440+0000 2024-03-25T18:02:09.308 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:02:09.308 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:09.308 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:02:09.308 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:02:09.308 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:02:09.308 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:09.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.306+0000 7f3ad044e640 1 Processor -- start 2024-03-25T18:02:09.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.306+0000 7f3ad044e640 1 -- start start 2024-03-25T18:02:09.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.306+0000 7f3ad044e640 10 monclient: init 2024-03-25T18:02:09.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.307+0000 7f3ad044e640 10 monclient: _reopen_session rank -1 2024-03-25T18:02:09.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.307+0000 7f3ad044e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:02:09.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.307+0000 7f3ad044e640 10 monclient(hunting): picked mon.noname-b con 0x7f3ac80a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:02:09.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.307+0000 7f3ad044e640 10 monclient(hunting): picked mon.noname-c con 0x7f3ac813f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:02:09.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.307+0000 7f3ad044e640 10 monclient(hunting): picked mon.noname-a con 0x7f3ac81430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:02:09.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.307+0000 7f3ad044e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac8142520 con 0x7f3ac81430c0 2024-03-25T18:02:09.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.307+0000 7f3ad044e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac80b28a0 con 0x7f3ac813f710 2024-03-25T18:02:09.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.307+0000 7f3ad044e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac80b2a30 con 0x7f3ac80a98f0 2024-03-25T18:02:09.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.307+0000 7f3ad044e640 10 monclient(hunting): _renew_subs 2024-03-25T18:02:09.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.307+0000 7f3ad044e640 10 monclient(hunting): authenticate will time out at 7257.886470s 2024-03-25T18:02:09.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.310+0000 7f3ace1ea640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f3ac80a98f0 0x7f3ac80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:40008/0 (socket says 172.21.15.18:40008) 2024-03-25T18:02:09.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.311+0000 7f3ace1ea640 1 -- 172.21.15.18:0/669498425 learned_addr learned my addr 172.21.15.18:0/669498425 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:02:09.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.324+0000 7f3acd1e8640 1 -- 172.21.15.18:0/669498425 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870281820 0 0) 0x7f3ac80b2a30 con 0x7f3ac80a98f0 2024-03-25T18:02:09.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.325+0000 7f3acd1e8640 10 monclient(hunting): _negotiate 2024-03-25T18:02:09.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.325+0000 7f3acd1e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:09.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.325+0000 7f3acd1e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:09.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.325+0000 7f3acd1e8640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:02:09.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.325+0000 7f3acd1e8640 1 -- 172.21.15.18:0/669498425 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ab4003520 con 0x7f3ac80a98f0 2024-03-25T18:02:09.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.331+0000 7f3acd1e8640 1 -- 172.21.15.18:0/669498425 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1865407767 0 0) 0x7f3ab4003520 con 0x7f3ac80a98f0 2024-03-25T18:02:09.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.331+0000 7f3acd1e8640 1 -- 172.21.15.18:0/669498425 >> v1:172.21.15.169:6789/0 conn(0x7f3ac813f710 legacy=0x7f3ac81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:09.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.331+0000 7f3acd1e8640 1 -- 172.21.15.18:0/669498425 >> v1:172.21.15.18:6789/0 conn(0x7f3ac81430c0 legacy=0x7f3ac814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:09.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.331+0000 7f3acd1e8640 10 monclient: _finish_hunting 0 2024-03-25T18:02:09.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.331+0000 7f3acd1e8640 1 monclient: found mon.noname-b 2024-03-25T18:02:09.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.331+0000 7f3acd1e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:02:09.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.331+0000 7f3acd1e8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:02:09.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.331+0000 7f3acd1e8640 1 -- 172.21.15.18:0/669498425 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac80b44c0 con 0x7f3ac80a98f0 2024-03-25T18:02:09.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.331+0000 7f3acd1e8640 10 monclient: _finish_auth 0 2024-03-25T18:02:09.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.331+0000 7f3acd1e8640 10 monclient: _check_auth_tickets 2024-03-25T18:02:09.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.331+0000 7f3acd1e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:02:09.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.331+0000 7f3acd1e8640 1 -- 172.21.15.18:0/669498425 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3ac4001690 con 0x7f3ac80a98f0 2024-03-25T18:02:09.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.331+0000 7f3acd1e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:09.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.331+0000 7f3acd1e8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:02:09.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.331+0000 7f3acd1e8640 10 monclient: dump: 2024-03-25T18:02:09.333 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:09.334 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:09.334 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:09.334 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:09.334 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:09.334 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:09.334 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:09.334 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:09.334 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:09.334 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:09.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.331+0000 7f3ad044e640 5 monclient: authenticate success, global_id 4493 2024-03-25T18:02:09.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.331+0000 7f3ad044e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:02:09.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.339+0000 7f3acd1e8640 1 -- 172.21.15.18:0/669498425 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ac40017f0 con 0x7f3ac80a98f0 2024-03-25T18:02:09.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.339+0000 7f3acd1e8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:02:09.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.339+0000 7f3acd1e8640 1 -- 172.21.15.18:0/669498425 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3ac4001c00 con 0x7f3ac80a98f0 2024-03-25T18:02:09.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.339+0000 7f3ad044e640 10 monclient: get_monmap_and_config success 2024-03-25T18:02:09.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.339+0000 7f3ad044e640 4 set_mon_vals no callback set 2024-03-25T18:02:09.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.339+0000 7f3acd1e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:09.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.339+0000 7f3acd1e8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:02:09.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.339+0000 7f3acd1e8640 10 monclient: dump: 2024-03-25T18:02:09.340 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:09.341 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:09.341 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:09.341 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:09.341 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:09.341 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:09.341 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:09.341 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:09.341 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:09.341 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:09.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.340+0000 7f3ad044e640 10 monclient: shutdown 2024-03-25T18:02:09.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.340+0000 7f3ad044e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:02:09.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.340+0000 7f3ad044e640 1 -- 172.21.15.18:0/669498425 >> v1:172.21.15.18:6790/0 conn(0x7f3ac80a98f0 legacy=0x7f3ac80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:09.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.340+0000 7f3ad044e640 1 -- 172.21.15.18:0/669498425 shutdown_connections 2024-03-25T18:02:09.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.340+0000 7f3ad044e640 1 -- 172.21.15.18:0/669498425 >> 172.21.15.18:0/669498425 conn(0x7f3ac80b7ea0 msgr2=0x7f3ac80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:02:09.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.340+0000 7f3ad044e640 1 -- 172.21.15.18:0/669498425 shutdown_connections 2024-03-25T18:02:09.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.340+0000 7f3ad044e640 1 -- 172.21.15.18:0/669498425 wait complete. 2024-03-25T18:02:09.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.341+0000 7f3ad044e640 10 monclient: build_initial_monmap 2024-03-25T18:02:09.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.341+0000 7f3ad044e640 10 monclient: monmap: 2024-03-25T18:02:09.344 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:02:09.344 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:02:09.344 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:02:09.344 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:02:09.344 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:02:09.344 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:02:09.344 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:02:09.344 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:02:09.344 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:02:09.344 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:09.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.341+0000 7f3ad044e640 1 librados: starting msgr at 2024-03-25T18:02:09.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.341+0000 7f3ad044e640 1 librados: starting objecter 2024-03-25T18:02:09.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.341+0000 7f3ad044e640 1 Processor -- start 2024-03-25T18:02:09.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.341+0000 7f3ad044e640 1 -- start start 2024-03-25T18:02:09.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.341+0000 7f3ad044e640 1 librados: setting wanted keys 2024-03-25T18:02:09.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.341+0000 7f3ad044e640 1 librados: calling monclient init 2024-03-25T18:02:09.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.341+0000 7f3ad044e640 10 monclient: init 2024-03-25T18:02:09.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.342+0000 7f3ad044e640 10 monclient: _reopen_session rank -1 2024-03-25T18:02:09.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.342+0000 7f3ad044e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:02:09.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.342+0000 7f3ad044e640 10 monclient(hunting): picked mon.noname-a con 0x7f3ac80a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:02:09.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.342+0000 7f3ad044e640 10 monclient(hunting): picked mon.noname-b con 0x7f3ac813f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:02:09.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.342+0000 7f3ad044e640 10 monclient(hunting): picked mon.noname-c con 0x7f3ac81430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:02:09.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.342+0000 7f3ad044e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac8139a40 con 0x7f3ac80a98f0 2024-03-25T18:02:09.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.342+0000 7f3ad044e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac8133f20 con 0x7f3ac813f710 2024-03-25T18:02:09.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.342+0000 7f3ad044e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac80ab950 con 0x7f3ac81430c0 2024-03-25T18:02:09.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.342+0000 7f3ad044e640 10 monclient(hunting): _renew_subs 2024-03-25T18:02:09.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.342+0000 7f3ad044e640 10 monclient(hunting): authenticate will time out at 7257.921191s 2024-03-25T18:02:09.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.351+0000 7f3ace1ea640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f3ac80a98f0 0x7f3ac8138c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:48742/0 (socket says 172.21.15.18:48742) 2024-03-25T18:02:09.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.351+0000 7f3ace1ea640 1 -- 172.21.15.18:0/1331400457 learned_addr learned my addr 172.21.15.18:0/1331400457 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:02:09.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.351+0000 7f3abeffd640 10 client.?.objecter ms_handle_connect 0x7f3ac81430c0 2024-03-25T18:02:09.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.351+0000 7f3abeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:09.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.352+0000 7f3abeffd640 10 client.?.objecter ms_handle_connect 0x7f3ac80a98f0 2024-03-25T18:02:09.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.352+0000 7f3abeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:09.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.361+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356289910 0 0) 0x7f3ac80ab950 con 0x7f3ac81430c0 2024-03-25T18:02:09.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.361+0000 7f3abeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:02:09.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.361+0000 7f3abeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:09.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.361+0000 7f3abeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:09.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.361+0000 7f3abeffd640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:02:09.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.361+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aac004eb0 con 0x7f3ac81430c0 2024-03-25T18:02:09.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.364+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025800705 0 0) 0x7f3ac8139a40 con 0x7f3ac80a98f0 2024-03-25T18:02:09.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.364+0000 7f3abeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:02:09.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.364+0000 7f3abeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:09.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.364+0000 7f3abeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:09.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.364+0000 7f3abeffd640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:02:09.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.364+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ac80ab950 con 0x7f3ac80a98f0 2024-03-25T18:02:09.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.375+0000 7f3abeffd640 10 client.?.objecter ms_handle_connect 0x7f3ac813f710 2024-03-25T18:02:09.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.375+0000 7f3abeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:09.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.375+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3130603539 0 0) 0x7f3aac004eb0 con 0x7f3ac81430c0 2024-03-25T18:02:09.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.376+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ac8139a40 con 0x7f3ac81430c0 2024-03-25T18:02:09.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.376+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3ac0001b90 con 0x7f3ac81430c0 2024-03-25T18:02:09.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.376+0000 7f3abeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:09.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.376+0000 7f3abeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:02:09.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.376+0000 7f3abeffd640 10 monclient(hunting): dump: 2024-03-25T18:02:09.377 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:09.377 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:09.377 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:09.378 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:09.378 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:09.378 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:09.378 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:09.378 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:09.378 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:09.378 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:09.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.376+0000 7f3abeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:02:09.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.382+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3970208495 0 0) 0x7f3ac80ab950 con 0x7f3ac80a98f0 2024-03-25T18:02:09.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.382+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3aac004eb0 con 0x7f3ac80a98f0 2024-03-25T18:02:09.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.382+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3ac4001c00 con 0x7f3ac80a98f0 2024-03-25T18:02:09.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.382+0000 7f3abeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:09.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.382+0000 7f3abeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:02:09.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.382+0000 7f3abeffd640 10 monclient(hunting): dump: 2024-03-25T18:02:09.384 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:09.384 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:09.384 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:09.384 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:09.384 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:09.384 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:09.384 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:09.384 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:09.384 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:09.384 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:09.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1733253542 0 0) 0x7f3ac8139a40 con 0x7f3ac81430c0 2024-03-25T18:02:09.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 >> v1:172.21.15.169:6789/0 conn(0x7f3ac813f710 legacy=0x7f3ac8133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:09.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 >> v1:172.21.15.18:6789/0 conn(0x7f3ac80a98f0 legacy=0x7f3ac8138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:09.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3abeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:02:09.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3abeffd640 1 monclient: found mon.c 2024-03-25T18:02:09.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3abeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:02:09.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3abeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:02:09.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac80abc30 con 0x7f3ac81430c0 2024-03-25T18:02:09.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3abeffd640 10 monclient: _finish_auth 0 2024-03-25T18:02:09.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3abeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:02:09.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3abeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:02:09.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3ad044e640 5 monclient: authenticate success, global_id 4496 2024-03-25T18:02:09.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3ad044e640 10 monclient: _renew_subs 2024-03-25T18:02:09.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3ad044e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:02:09.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3ad044e640 1 -- 172.21.15.18:0/1331400457 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac80abf10 con 0x7f3ac81430c0 2024-03-25T18:02:09.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3ad044e640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:02:09.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3ad044e640 10 monclient: _renew_subs 2024-03-25T18:02:09.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3ad044e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:02:09.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3ad044e640 1 -- 172.21.15.18:0/1331400457 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac8139370 con 0x7f3ac81430c0 2024-03-25T18:02:09.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.384+0000 7f3ad044e640 1 librados: init done 2024-03-25T18:02:09.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.385+0000 7f3ad044e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:09.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.385+0000 7f3ad044e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:09.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.385+0000 7f3ad044e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:02:09.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.385+0000 7f3ad044e640 1 -- 172.21.15.18:0/1331400457 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac8138980 con 0x7f3ac81430c0 2024-03-25T18:02:09.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.694+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ac0002080 con 0x7f3ac81430c0 2024-03-25T18:02:09.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.694+0000 7f3abeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:02:09.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.694+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3ac0003b10 con 0x7f3ac81430c0 2024-03-25T18:02:09.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.694+0000 7f3abeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:09.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.694+0000 7f3abeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:02:09.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.694+0000 7f3abeffd640 10 monclient: dump: 2024-03-25T18:02:09.695 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:09.696 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:09.696 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:09.696 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:09.696 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:09.696 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:09.696 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:09.696 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:09.696 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:09.696 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:09.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.694+0000 7f3acc9e7640 4 set_mon_vals no callback set 2024-03-25T18:02:09.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.695+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f3ac0003d30 con 0x7f3ac81430c0 2024-03-25T18:02:09.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.695+0000 7f3ace1ea640 10 monclient: get_auth_request con 0x7f3aac06de40 auth_method 0 2024-03-25T18:02:09.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.696+0000 7f3abeffd640 10 client.4496.objecter ms_handle_connect 0x7f3aac06de40 2024-03-25T18:02:09.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.704+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f3ac00538b0 con 0x7f3ac81430c0 2024-03-25T18:02:09.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.704+0000 7f3abeffd640 10 client.4496.objecter ms_dispatch 0x7f3ac8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:02:09.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.704+0000 7f3abeffd640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:02:09.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.704+0000 7f3abeffd640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:02:09.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.704+0000 7f3abeffd640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-25T18:02:09.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.704+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3ac004f1f0 con 0x7f3ac81430c0 2024-03-25T18:02:09.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.704+0000 7f3abeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:09.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.704+0000 7f3abeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:02:09.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.968+0000 7f3ad044e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:09.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.968+0000 7f3ad044e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:09.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.968+0000 7f3ad044e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:02:09.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.968+0000 7f3ad044e640 1 -- 172.21.15.18:0/1331400457 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ac80627b0 con 0x7f3ac81430c0 2024-03-25T18:02:09.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.992+0000 7f3abeffd640 1 -- 172.21.15.18:0/1331400457 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f3ac00512f0 con 0x7f3ac81430c0 2024-03-25T18:02:09.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.992+0000 7f3abeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:09.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.992+0000 7f3abeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:02:09.994 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:02:09.994 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:02:09.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.995+0000 7f3ad044e640 10 librados: watch_flush enter 2024-03-25T18:02:09.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.995+0000 7f3ad044e640 10 librados: watch_flush exit 2024-03-25T18:02:09.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.995+0000 7f3ad044e640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-25T18:02:09.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.995+0000 7f3ad044e640 10 client.4496.objecter successfully canceled tick 2024-03-25T18:02:09.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.995+0000 7f3ad044e640 1 -- 172.21.15.18:0/1331400457 >> v1:172.21.15.18:6800/3721244451 conn(0x7f3aac06de40 legacy=0x7f3aac070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:09.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.995+0000 7f3ad044e640 10 monclient: shutdown 2024-03-25T18:02:09.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.995+0000 7f3ad044e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:02:09.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.995+0000 7f3ad044e640 1 -- 172.21.15.18:0/1331400457 >> v1:172.21.15.18:6790/0 conn(0x7f3ac81430c0 legacy=0x7f3ac8133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:09.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.995+0000 7f3ad044e640 1 -- 172.21.15.18:0/1331400457 shutdown_connections 2024-03-25T18:02:09.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.996+0000 7f3ad044e640 1 -- 172.21.15.18:0/1331400457 >> 172.21.15.18:0/1331400457 conn(0x7f3ac80b7ea0 msgr2=0x7f3ac8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:02:09.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.996+0000 7f3ad044e640 1 -- 172.21.15.18:0/1331400457 shutdown_connections 2024-03-25T18:02:09.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.996+0000 7f3ad044e640 1 -- 172.21.15.18:0/1331400457 wait complete. 2024-03-25T18:02:09.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:09.996+0000 7f3ad044e640 1 librados: shutdown 2024-03-25T18:02:10.007 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:02:16.813 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:02:16.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.930+0000 7f57e67c8640 10 monclient: get_monmap_and_config 2024-03-25T18:02:16.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.930+0000 7f57e67c8640 10 monclient: build_initial_monmap 2024-03-25T18:02:16.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.930+0000 7f57e67c8640 10 monclient: monmap: 2024-03-25T18:02:16.931 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:02:16.931 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:16.931 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:02:16.931407+0000 2024-03-25T18:02:16.931 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:02:16.931407+0000 2024-03-25T18:02:16.932 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:02:16.932 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:16.932 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:02:16.932 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:02:16.932 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:02:16.932 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:16.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.930+0000 7f57e67c8640 1 Processor -- start 2024-03-25T18:02:16.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.930+0000 7f57e67c8640 1 -- start start 2024-03-25T18:02:16.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.930+0000 7f57e67c8640 10 monclient: init 2024-03-25T18:02:16.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.931+0000 7f57e67c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:02:16.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.931+0000 7f57e67c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:02:16.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.931+0000 7f57e67c8640 10 monclient(hunting): picked mon.noname-b con 0x7f57e00a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:02:16.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.931+0000 7f57e67c8640 10 monclient(hunting): picked mon.noname-a con 0x7f57e013f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:02:16.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.931+0000 7f57e67c8640 10 monclient(hunting): picked mon.noname-c con 0x7f57e01430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:02:16.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.931+0000 7f57e67c8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e0142520 con 0x7f57e013f710 2024-03-25T18:02:16.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.931+0000 7f57e67c8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e00b28a0 con 0x7f57e01430c0 2024-03-25T18:02:16.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.931+0000 7f57e67c8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e00b2a30 con 0x7f57e00a98f0 2024-03-25T18:02:16.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.931+0000 7f57e67c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:02:16.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.931+0000 7f57e67c8640 10 monclient(hunting): authenticate will time out at 7265.510576s 2024-03-25T18:02:16.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.946+0000 7f57df7fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f57e013f710 0x7f57e01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:44538/0 (socket says 172.21.15.18:44538) 2024-03-25T18:02:16.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.946+0000 7f57df7fe640 1 -- 172.21.15.18:0/3009342637 learned_addr learned my addr 172.21.15.18:0/3009342637 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:02:16.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.960+0000 7f57deffd640 1 -- 172.21.15.18:0/3009342637 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433837095 0 0) 0x7f57e00b28a0 con 0x7f57e01430c0 2024-03-25T18:02:16.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.960+0000 7f57deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:02:16.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.960+0000 7f57deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:16.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.960+0000 7f57deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:16.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.960+0000 7f57deffd640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:02:16.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.960+0000 7f57deffd640 1 -- 172.21.15.18:0/3009342637 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57c8003520 con 0x7f57e01430c0 2024-03-25T18:02:16.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.970+0000 7f57deffd640 1 -- 172.21.15.18:0/3009342637 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659051964 0 0) 0x7f57e00b2a30 con 0x7f57e00a98f0 2024-03-25T18:02:16.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.970+0000 7f57deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:02:16.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.970+0000 7f57deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:16.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.970+0000 7f57deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:16.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.970+0000 7f57deffd640 10 monclient(hunting): my global_id is 4499 2024-03-25T18:02:16.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.970+0000 7f57deffd640 1 -- 172.21.15.18:0/3009342637 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e00b28a0 con 0x7f57e00a98f0 2024-03-25T18:02:16.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.972+0000 7f57deffd640 1 -- 172.21.15.18:0/3009342637 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758264189 0 0) 0x7f57e0142520 con 0x7f57e013f710 2024-03-25T18:02:16.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.972+0000 7f57deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:02:16.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.972+0000 7f57deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:16.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.972+0000 7f57deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:16.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.972+0000 7f57deffd640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:02:16.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.972+0000 7f57deffd640 1 -- 172.21.15.18:0/3009342637 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e00b2a30 con 0x7f57e013f710 2024-03-25T18:02:16.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.975+0000 7f57deffd640 1 -- 172.21.15.18:0/3009342637 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2030404446 0 0) 0x7f57c8003520 con 0x7f57e01430c0 2024-03-25T18:02:16.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.975+0000 7f57deffd640 1 -- 172.21.15.18:0/3009342637 >> v1:172.21.15.18:6790/0 conn(0x7f57e00a98f0 legacy=0x7f57e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:16.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.975+0000 7f57deffd640 1 -- 172.21.15.18:0/3009342637 >> v1:172.21.15.18:6789/0 conn(0x7f57e013f710 legacy=0x7f57e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:16.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.975+0000 7f57deffd640 10 monclient: _finish_hunting 0 2024-03-25T18:02:16.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.975+0000 7f57deffd640 1 monclient: found mon.noname-c 2024-03-25T18:02:16.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.975+0000 7f57deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:02:16.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.975+0000 7f57deffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:02:16.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.975+0000 7f57deffd640 1 -- 172.21.15.18:0/3009342637 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e00b44c0 con 0x7f57e01430c0 2024-03-25T18:02:16.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.975+0000 7f57deffd640 10 monclient: _finish_auth 0 2024-03-25T18:02:16.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.975+0000 7f57deffd640 10 monclient: _check_auth_tickets 2024-03-25T18:02:16.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.975+0000 7f57deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:02:16.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.975+0000 7f57deffd640 1 -- 172.21.15.18:0/3009342637 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f57d8001a10 con 0x7f57e01430c0 2024-03-25T18:02:16.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.975+0000 7f57deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:16.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.975+0000 7f57deffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:02:16.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.975+0000 7f57deffd640 10 monclient: dump: 2024-03-25T18:02:16.977 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:16.978 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:16.978 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:16.978 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:16.978 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:16.978 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:16.978 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:16.978 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:16.978 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:16.978 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:16.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.975+0000 7f57e67c8640 5 monclient: authenticate success, global_id 4480 2024-03-25T18:02:16.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.975+0000 7f57e67c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:02:16.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.984+0000 7f57deffd640 1 -- 172.21.15.18:0/3009342637 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57d8001b70 con 0x7f57e01430c0 2024-03-25T18:02:16.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.984+0000 7f57deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:02:16.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.984+0000 7f57deffd640 1 -- 172.21.15.18:0/3009342637 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f57d8001f80 con 0x7f57e01430c0 2024-03-25T18:02:16.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.984+0000 7f57deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:16.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.984+0000 7f57deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:02:16.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.984+0000 7f57deffd640 10 monclient: dump: 2024-03-25T18:02:16.986 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:16.986 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:16.986 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:16.986 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:16.986 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:16.986 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:16.986 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:16.986 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:16.986 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:16.987 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:16.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.984+0000 7f57e67c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:02:16.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.984+0000 7f57e67c8640 4 set_mon_vals no callback set 2024-03-25T18:02:16.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.985+0000 7f57e67c8640 10 monclient: shutdown 2024-03-25T18:02:16.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.985+0000 7f57e67c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:02:16.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.985+0000 7f57e67c8640 1 -- 172.21.15.18:0/3009342637 >> v1:172.21.15.169:6789/0 conn(0x7f57e01430c0 legacy=0x7f57e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:16.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.985+0000 7f57e67c8640 1 -- 172.21.15.18:0/3009342637 shutdown_connections 2024-03-25T18:02:16.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.985+0000 7f57e67c8640 1 -- 172.21.15.18:0/3009342637 >> 172.21.15.18:0/3009342637 conn(0x7f57e00b7ea0 msgr2=0x7f57e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:02:16.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.985+0000 7f57e67c8640 1 -- 172.21.15.18:0/3009342637 shutdown_connections 2024-03-25T18:02:16.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.985+0000 7f57e67c8640 1 -- 172.21.15.18:0/3009342637 wait complete. 2024-03-25T18:02:16.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.986+0000 7f57e67c8640 10 monclient: build_initial_monmap 2024-03-25T18:02:16.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.986+0000 7f57e67c8640 10 monclient: monmap: 2024-03-25T18:02:16.989 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:02:16.989 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:02:16.989 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:02:16.989 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:02:16.989 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:02:16.989 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:02:16.989 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:02:16.989 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:02:16.989 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:02:16.989 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:16.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.986+0000 7f57e67c8640 1 librados: starting msgr at 2024-03-25T18:02:16.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.986+0000 7f57e67c8640 1 librados: starting objecter 2024-03-25T18:02:16.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.986+0000 7f57e67c8640 1 Processor -- start 2024-03-25T18:02:16.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.986+0000 7f57e67c8640 1 -- start start 2024-03-25T18:02:16.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.986+0000 7f57e67c8640 1 librados: setting wanted keys 2024-03-25T18:02:16.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.986+0000 7f57e67c8640 1 librados: calling monclient init 2024-03-25T18:02:16.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.986+0000 7f57e67c8640 10 monclient: init 2024-03-25T18:02:16.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.987+0000 7f57e67c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:02:16.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.987+0000 7f57e67c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:02:16.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.987+0000 7f57e67c8640 10 monclient(hunting): picked mon.noname-c con 0x7f57e00a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:02:16.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.987+0000 7f57e67c8640 10 monclient(hunting): picked mon.noname-b con 0x7f57e013f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:02:16.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.987+0000 7f57e67c8640 10 monclient(hunting): picked mon.noname-a con 0x7f57e01430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:02:16.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.987+0000 7f57e67c8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e0135230 con 0x7f57e01430c0 2024-03-25T18:02:16.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.987+0000 7f57e67c8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e0135da0 con 0x7f57e013f710 2024-03-25T18:02:16.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.987+0000 7f57e67c8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e0135f30 con 0x7f57e00a98f0 2024-03-25T18:02:16.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.987+0000 7f57e67c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:02:16.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.987+0000 7f57e67c8640 10 monclient(hunting): authenticate will time out at 7265.566279s 2024-03-25T18:02:16.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.991+0000 7f57e4d65640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f57e01430c0 0x7f57e01db560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:44578/0 (socket says 172.21.15.18:44578) 2024-03-25T18:02:16.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.991+0000 7f57e4d65640 1 -- 172.21.15.18:0/1200056715 learned_addr learned my addr 172.21.15.18:0/1200056715 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:02:16.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.992+0000 7f57dcff9640 10 client.?.objecter ms_handle_connect 0x7f57e01430c0 2024-03-25T18:02:16.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.992+0000 7f57dcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:16.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.992+0000 7f57dcff9640 10 client.?.objecter ms_handle_connect 0x7f57e013f710 2024-03-25T18:02:16.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.992+0000 7f57dcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:16.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.996+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633568938 0 0) 0x7f57e0135230 con 0x7f57e01430c0 2024-03-25T18:02:16.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.996+0000 7f57dcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:02:16.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.996+0000 7f57dcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:16.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.996+0000 7f57dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:16.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.996+0000 7f57dcff9640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:02:16.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:16.996+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57cc004eb0 con 0x7f57e01430c0 2024-03-25T18:02:17.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.003+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2750544582 0 0) 0x7f57cc004eb0 con 0x7f57e01430c0 2024-03-25T18:02:17.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.003+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57e0135230 con 0x7f57e01430c0 2024-03-25T18:02:17.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.004+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812088030 0 0) 0x7f57e0135da0 con 0x7f57e013f710 2024-03-25T18:02:17.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.004+0000 7f57dcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:02:17.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.004+0000 7f57dcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:17.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.004+0000 7f57dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:17.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.004+0000 7f57dcff9640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:02:17.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.004+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57cc004eb0 con 0x7f57e013f710 2024-03-25T18:02:17.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.007+0000 7f57dcff9640 10 client.?.objecter ms_handle_connect 0x7f57e00a98f0 2024-03-25T18:02:17.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.007+0000 7f57dcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:17.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.017+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 375936781 0 0) 0x7f57cc004eb0 con 0x7f57e013f710 2024-03-25T18:02:17.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.017+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57e0135da0 con 0x7f57e013f710 2024-03-25T18:02:17.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.017+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f57d4001830 con 0x7f57e013f710 2024-03-25T18:02:17.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.017+0000 7f57dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:17.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.017+0000 7f57dcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:02:17.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.017+0000 7f57dcff9640 10 monclient(hunting): dump: 2024-03-25T18:02:17.019 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:17.019 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:17.019 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:17.019 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:17.019 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:17.019 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:17.019 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:17.019 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:17.019 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:17.019 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:17.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.017+0000 7f57dcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:02:17.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f57d8002000 con 0x7f57e01430c0 2024-03-25T18:02:17.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:17.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57dcff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:02:17.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57dcff9640 10 monclient(hunting): dump: 2024-03-25T18:02:17.025 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:17.026 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:17.026 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:17.026 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:17.026 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:17.026 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:17.026 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:17.026 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:17.026 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:17.026 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:17.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4039161144 0 0) 0x7f57e0135230 con 0x7f57e01430c0 2024-03-25T18:02:17.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 >> v1:172.21.15.18:6790/0 conn(0x7f57e00a98f0 legacy=0x7f57e01d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:17.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 >> v1:172.21.15.169:6789/0 conn(0x7f57e013f710 legacy=0x7f57e01dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:17.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57dcff9640 10 monclient: _finish_hunting 0 2024-03-25T18:02:17.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57dcff9640 1 monclient: found mon.a 2024-03-25T18:02:17.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57dcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:02:17.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57dcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:02:17.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e01360c0 con 0x7f57e01430c0 2024-03-25T18:02:17.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57dcff9640 10 monclient: _finish_auth 0 2024-03-25T18:02:17.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57dcff9640 10 monclient: _check_auth_tickets 2024-03-25T18:02:17.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:02:17.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57e67c8640 5 monclient: authenticate success, global_id 4494 2024-03-25T18:02:17.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57e67c8640 10 monclient: _renew_subs 2024-03-25T18:02:17.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57e67c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:02:17.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57e67c8640 1 -- 172.21.15.18:0/1200056715 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e01d5000 con 0x7f57e01430c0 2024-03-25T18:02:17.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57e67c8640 10 client.4494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:02:17.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57e67c8640 10 monclient: _renew_subs 2024-03-25T18:02:17.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57e67c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:02:17.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.023+0000 7f57e67c8640 1 -- 172.21.15.18:0/1200056715 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e01d41e0 con 0x7f57e01430c0 2024-03-25T18:02:17.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.024+0000 7f57e67c8640 1 librados: init done 2024-03-25T18:02:17.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.025+0000 7f57e67c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:17.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.025+0000 7f57e67c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:17.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.025+0000 7f57e67c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:02:17.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.025+0000 7f57e67c8640 1 -- 172.21.15.18:0/1200056715 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e0135090 con 0x7f57e01430c0 2024-03-25T18:02:17.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.301+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57d8001510 con 0x7f57e01430c0 2024-03-25T18:02:17.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.301+0000 7f57dcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:02:17.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.301+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f57d8003ca0 con 0x7f57e01430c0 2024-03-25T18:02:17.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.301+0000 7f57dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:17.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.301+0000 7f57dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:02:17.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.301+0000 7f57dcff9640 10 monclient: dump: 2024-03-25T18:02:17.303 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:17.303 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:17.303 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:17.303 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:17.303 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:17.304 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:17.304 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:17.304 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:17.304 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:17.304 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:17.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.301+0000 7f57de7fc640 4 set_mon_vals no callback set 2024-03-25T18:02:17.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.302+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f57d8003e80 con 0x7f57e01430c0 2024-03-25T18:02:17.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.302+0000 7f57dffff640 10 monclient: get_auth_request con 0x7f57cc06df10 auth_method 0 2024-03-25T18:02:17.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.303+0000 7f57dcff9640 10 client.4494.objecter ms_handle_connect 0x7f57cc06df10 2024-03-25T18:02:17.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.307+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f57d8054a50 con 0x7f57e01430c0 2024-03-25T18:02:17.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.307+0000 7f57dcff9640 10 client.4494.objecter ms_dispatch 0x7f57e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:02:17.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.307+0000 7f57dcff9640 3 client.4494.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:02:17.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.307+0000 7f57dcff9640 3 client.4494.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:02:17.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.308+0000 7f57dcff9640 20 client.4494.objecter dump_active .. 0 homeless 2024-03-25T18:02:17.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.308+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f57d8050560 con 0x7f57e01430c0 2024-03-25T18:02:17.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.308+0000 7f57dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:17.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.308+0000 7f57dcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:02:17.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.539+0000 7f57e67c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:17.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.539+0000 7f57e67c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:17.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.539+0000 7f57e67c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:02:17.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.539+0000 7f57e67c8640 1 -- 172.21.15.18:0/1200056715 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57e01353c0 con 0x7f57e01430c0 2024-03-25T18:02:17.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.569+0000 7f57dcff9640 1 -- 172.21.15.18:0/1200056715 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f57d8052490 con 0x7f57e01430c0 2024-03-25T18:02:17.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.569+0000 7f57dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:17.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.569+0000 7f57dcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:02:17.571 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:02:17.571 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:02:17.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.571+0000 7f57e67c8640 10 librados: watch_flush enter 2024-03-25T18:02:17.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.572+0000 7f57e67c8640 10 librados: watch_flush exit 2024-03-25T18:02:17.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.572+0000 7f57e67c8640 20 client.4494.objecter shutdown clearing up homeless session... 2024-03-25T18:02:17.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.572+0000 7f57e67c8640 10 client.4494.objecter successfully canceled tick 2024-03-25T18:02:17.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.572+0000 7f57e67c8640 1 -- 172.21.15.18:0/1200056715 >> v1:172.21.15.18:6800/3721244451 conn(0x7f57cc06df10 legacy=0x7f57cc0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:17.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.572+0000 7f57e67c8640 10 monclient: shutdown 2024-03-25T18:02:17.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.572+0000 7f57e67c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:02:17.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.572+0000 7f57e67c8640 1 -- 172.21.15.18:0/1200056715 >> v1:172.21.15.18:6789/0 conn(0x7f57e01430c0 legacy=0x7f57e01db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:17.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.572+0000 7f57e67c8640 1 -- 172.21.15.18:0/1200056715 shutdown_connections 2024-03-25T18:02:17.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.572+0000 7f57e67c8640 1 -- 172.21.15.18:0/1200056715 >> 172.21.15.18:0/1200056715 conn(0x7f57e00b7ea0 msgr2=0x7f57e014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:02:17.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.572+0000 7f57e67c8640 1 -- 172.21.15.18:0/1200056715 shutdown_connections 2024-03-25T18:02:17.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.572+0000 7f57e67c8640 1 -- 172.21.15.18:0/1200056715 wait complete. 2024-03-25T18:02:17.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:17.572+0000 7f57e67c8640 1 librados: shutdown 2024-03-25T18:02:17.585 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:02:23.116 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:06:20.909 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:23.133 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:06:20.926 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:23.160 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:06:20.953 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:23.183 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:20.932 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:23.191 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:20.940 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:23.206 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:06:20.998 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:23.211 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:06:21.004 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:23.215 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:20.964 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:23.229 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:06:21.022 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:24.391 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:02:24.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.511+0000 7f0c723c4640 10 monclient: get_monmap_and_config 2024-03-25T18:02:24.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.511+0000 7f0c723c4640 10 monclient: build_initial_monmap 2024-03-25T18:02:24.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.511+0000 7f0c723c4640 10 monclient: monmap: 2024-03-25T18:02:24.512 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:02:24.513 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:24.513 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:02:24.512554+0000 2024-03-25T18:02:24.513 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:02:24.512554+0000 2024-03-25T18:02:24.513 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:02:24.513 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:24.513 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:02:24.513 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:02:24.513 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:02:24.513 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:24.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.512+0000 7f0c723c4640 1 Processor -- start 2024-03-25T18:02:24.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.512+0000 7f0c723c4640 1 -- start start 2024-03-25T18:02:24.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.512+0000 7f0c723c4640 10 monclient: init 2024-03-25T18:02:24.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.512+0000 7f0c723c4640 10 monclient: _reopen_session rank -1 2024-03-25T18:02:24.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.512+0000 7f0c723c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:02:24.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.512+0000 7f0c723c4640 10 monclient(hunting): picked mon.noname-b con 0x7f0c6c0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:02:24.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.513+0000 7f0c723c4640 10 monclient(hunting): picked mon.noname-c con 0x7f0c6c13f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:02:24.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.513+0000 7f0c723c4640 10 monclient(hunting): picked mon.noname-a con 0x7f0c6c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:02:24.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.513+0000 7f0c723c4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6c142520 con 0x7f0c6c1430c0 2024-03-25T18:02:24.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.513+0000 7f0c723c4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6c0b28a0 con 0x7f0c6c13f710 2024-03-25T18:02:24.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.513+0000 7f0c723c4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6c0b2a30 con 0x7f0c6c0a98f0 2024-03-25T18:02:24.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.513+0000 7f0c723c4640 10 monclient(hunting): _renew_subs 2024-03-25T18:02:24.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.513+0000 7f0c723c4640 10 monclient(hunting): authenticate will time out at 7273.091789s 2024-03-25T18:02:24.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.523+0000 7f0c6b7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f0c6c13f710 0x7f0c6c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:58082/0 (socket says 172.21.15.18:58082) 2024-03-25T18:02:24.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.523+0000 7f0c6b7fe640 1 -- 172.21.15.18:0/3686121174 learned_addr learned my addr 172.21.15.18:0/3686121174 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:02:24.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.533+0000 7f0c6affd640 1 -- 172.21.15.18:0/3686121174 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146525004 0 0) 0x7f0c6c0b28a0 con 0x7f0c6c13f710 2024-03-25T18:02:24.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.533+0000 7f0c6affd640 10 monclient(hunting): _negotiate 2024-03-25T18:02:24.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.533+0000 7f0c6affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:24.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.533+0000 7f0c6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:24.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.533+0000 7f0c6affd640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:02:24.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.533+0000 7f0c6affd640 1 -- 172.21.15.18:0/3686121174 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c54003520 con 0x7f0c6c13f710 2024-03-25T18:02:24.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.544+0000 7f0c6affd640 1 -- 172.21.15.18:0/3686121174 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006094759 0 0) 0x7f0c6c0b2a30 con 0x7f0c6c0a98f0 2024-03-25T18:02:24.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.544+0000 7f0c6affd640 10 monclient(hunting): _negotiate 2024-03-25T18:02:24.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.544+0000 7f0c6affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:24.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.544+0000 7f0c6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:24.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.544+0000 7f0c6affd640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:02:24.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.544+0000 7f0c6affd640 1 -- 172.21.15.18:0/3686121174 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c6c0b28a0 con 0x7f0c6c0a98f0 2024-03-25T18:02:24.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.547+0000 7f0c6affd640 1 -- 172.21.15.18:0/3686121174 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1072741146 0 0) 0x7f0c54003520 con 0x7f0c6c13f710 2024-03-25T18:02:24.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.547+0000 7f0c6affd640 1 -- 172.21.15.18:0/3686121174 >> v1:172.21.15.18:6790/0 conn(0x7f0c6c0a98f0 legacy=0x7f0c6c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:24.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.547+0000 7f0c6affd640 1 -- 172.21.15.18:0/3686121174 >> v1:172.21.15.18:6789/0 conn(0x7f0c6c1430c0 legacy=0x7f0c6c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:24.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.547+0000 7f0c6affd640 10 monclient: _finish_hunting 0 2024-03-25T18:02:24.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.547+0000 7f0c6affd640 1 monclient: found mon.noname-c 2024-03-25T18:02:24.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.547+0000 7f0c6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:02:24.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.547+0000 7f0c6affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:02:24.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.547+0000 7f0c6affd640 1 -- 172.21.15.18:0/3686121174 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c0b44c0 con 0x7f0c6c13f710 2024-03-25T18:02:24.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.547+0000 7f0c6affd640 10 monclient: _finish_auth 0 2024-03-25T18:02:24.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.547+0000 7f0c6affd640 10 monclient: _check_auth_tickets 2024-03-25T18:02:24.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.547+0000 7f0c6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:02:24.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.547+0000 7f0c6affd640 1 -- 172.21.15.18:0/3686121174 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0c60001690 con 0x7f0c6c13f710 2024-03-25T18:02:24.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.547+0000 7f0c6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:24.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.547+0000 7f0c6affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:02:24.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.547+0000 7f0c6affd640 10 monclient: dump: 2024-03-25T18:02:24.550 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:24.550 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:24.550 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:24.550 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:24.550 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:24.550 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:24.550 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:24.550 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:24.550 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:24.550 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:24.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.547+0000 7f0c723c4640 5 monclient: authenticate success, global_id 4486 2024-03-25T18:02:24.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.547+0000 7f0c723c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:02:24.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.560+0000 7f0c6affd640 1 -- 172.21.15.18:0/3686121174 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c600017f0 con 0x7f0c6c13f710 2024-03-25T18:02:24.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.560+0000 7f0c6affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:02:24.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.560+0000 7f0c6affd640 1 -- 172.21.15.18:0/3686121174 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0c60001c00 con 0x7f0c6c13f710 2024-03-25T18:02:24.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.560+0000 7f0c6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:24.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.560+0000 7f0c6affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:02:24.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.560+0000 7f0c6affd640 10 monclient: dump: 2024-03-25T18:02:24.561 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:24.561 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:24.561 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:24.561 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:24.562 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:24.562 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:24.562 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:24.562 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:24.562 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:24.562 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:24.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.560+0000 7f0c723c4640 10 monclient: get_monmap_and_config success 2024-03-25T18:02:24.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.560+0000 7f0c723c4640 4 set_mon_vals no callback set 2024-03-25T18:02:24.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.560+0000 7f0c723c4640 10 monclient: shutdown 2024-03-25T18:02:24.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.560+0000 7f0c723c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:02:24.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.560+0000 7f0c723c4640 1 -- 172.21.15.18:0/3686121174 >> v1:172.21.15.169:6789/0 conn(0x7f0c6c13f710 legacy=0x7f0c6c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:24.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.561+0000 7f0c723c4640 1 -- 172.21.15.18:0/3686121174 shutdown_connections 2024-03-25T18:02:24.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.561+0000 7f0c723c4640 1 -- 172.21.15.18:0/3686121174 >> 172.21.15.18:0/3686121174 conn(0x7f0c6c0b7ea0 msgr2=0x7f0c6c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:02:24.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.561+0000 7f0c723c4640 1 -- 172.21.15.18:0/3686121174 shutdown_connections 2024-03-25T18:02:24.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.561+0000 7f0c723c4640 1 -- 172.21.15.18:0/3686121174 wait complete. 2024-03-25T18:02:24.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.561+0000 7f0c723c4640 10 monclient: build_initial_monmap 2024-03-25T18:02:24.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.561+0000 7f0c723c4640 10 monclient: monmap: 2024-03-25T18:02:24.565 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:02:24.565 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:02:24.565 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:02:24.565 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:02:24.565 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:02:24.565 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:02:24.565 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:02:24.565 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:02:24.565 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:02:24.565 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:24.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.562+0000 7f0c723c4640 1 librados: starting msgr at 2024-03-25T18:02:24.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.562+0000 7f0c723c4640 1 librados: starting objecter 2024-03-25T18:02:24.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.562+0000 7f0c723c4640 1 Processor -- start 2024-03-25T18:02:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.562+0000 7f0c723c4640 1 -- start start 2024-03-25T18:02:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.562+0000 7f0c723c4640 1 librados: setting wanted keys 2024-03-25T18:02:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.562+0000 7f0c723c4640 1 librados: calling monclient init 2024-03-25T18:02:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.562+0000 7f0c723c4640 10 monclient: init 2024-03-25T18:02:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.562+0000 7f0c723c4640 10 monclient: _reopen_session rank -1 2024-03-25T18:02:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.563+0000 7f0c723c4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:02:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.563+0000 7f0c723c4640 10 monclient(hunting): picked mon.noname-a con 0x7f0c6c0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:02:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.563+0000 7f0c723c4640 10 monclient(hunting): picked mon.noname-b con 0x7f0c6c13f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:02:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.563+0000 7f0c723c4640 10 monclient(hunting): picked mon.noname-c con 0x7f0c6c1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:02:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.563+0000 7f0c723c4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6c1d77a0 con 0x7f0c6c0a98f0 2024-03-25T18:02:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.563+0000 7f0c723c4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6c138e40 con 0x7f0c6c13f710 2024-03-25T18:02:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.563+0000 7f0c723c4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6c0aba30 con 0x7f0c6c1430c0 2024-03-25T18:02:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.563+0000 7f0c723c4640 10 monclient(hunting): _renew_subs 2024-03-25T18:02:24.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.563+0000 7f0c723c4640 10 monclient(hunting): authenticate will time out at 7273.141836s 2024-03-25T18:02:24.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.580+0000 7f0c6bfff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f0c6c0a98f0 0x7f0c6c138730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:43742/0 (socket says 172.21.15.18:43742) 2024-03-25T18:02:24.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.580+0000 7f0c6bfff640 1 -- 172.21.15.18:0/1614327087 learned_addr learned my addr 172.21.15.18:0/1614327087 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:02:24.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.580+0000 7f0c68ff9640 10 client.?.objecter ms_handle_connect 0x7f0c6c0a98f0 2024-03-25T18:02:24.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.580+0000 7f0c68ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:24.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.584+0000 7f0c68ff9640 10 client.?.objecter ms_handle_connect 0x7f0c6c1430c0 2024-03-25T18:02:24.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.584+0000 7f0c68ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:24.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.587+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330151540 0 0) 0x7f0c6c1d77a0 con 0x7f0c6c0a98f0 2024-03-25T18:02:24.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.587+0000 7f0c68ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:02:24.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.587+0000 7f0c68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:24.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.587+0000 7f0c68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:24.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.587+0000 7f0c68ff9640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:02:24.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.587+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c58004eb0 con 0x7f0c6c0a98f0 2024-03-25T18:02:24.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.589+0000 7f0c68ff9640 10 client.?.objecter ms_handle_connect 0x7f0c6c13f710 2024-03-25T18:02:24.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.589+0000 7f0c68ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:24.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.591+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82988651 0 0) 0x7f0c6c0aba30 con 0x7f0c6c1430c0 2024-03-25T18:02:24.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.591+0000 7f0c68ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:02:24.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.591+0000 7f0c68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:24.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.591+0000 7f0c68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:24.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.591+0000 7f0c68ff9640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:02:24.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.591+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c6c1d77a0 con 0x7f0c6c1430c0 2024-03-25T18:02:24.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.598+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359861154 0 0) 0x7f0c6c138e40 con 0x7f0c6c13f710 2024-03-25T18:02:24.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.598+0000 7f0c68ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:02:24.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.598+0000 7f0c68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:24.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.598+0000 7f0c68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:24.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.598+0000 7f0c68ff9640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:02:24.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.598+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c6c0aba30 con 0x7f0c6c13f710 2024-03-25T18:02:24.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.601+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 331881744 0 0) 0x7f0c6c1d77a0 con 0x7f0c6c1430c0 2024-03-25T18:02:24.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.601+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c6c138e40 con 0x7f0c6c1430c0 2024-03-25T18:02:24.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.601+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0c64001b00 con 0x7f0c6c1430c0 2024-03-25T18:02:24.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.601+0000 7f0c68ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:24.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.602+0000 7f0c68ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:02:24.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.602+0000 7f0c68ff9640 10 monclient(hunting): dump: 2024-03-25T18:02:24.603 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:24.603 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:24.604 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:24.604 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:24.604 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:24.604 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:24.604 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:24.604 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:24.604 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:24.604 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:24.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.602+0000 7f0c68ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:02:24.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.606+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2624364692 0 0) 0x7f0c58004eb0 con 0x7f0c6c0a98f0 2024-03-25T18:02:24.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.606+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c6c1d77a0 con 0x7f0c6c0a98f0 2024-03-25T18:02:24.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.606+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0c5c001a90 con 0x7f0c6c0a98f0 2024-03-25T18:02:24.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.606+0000 7f0c68ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:24.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.606+0000 7f0c68ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:02:24.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.606+0000 7f0c68ff9640 10 monclient(hunting): dump: 2024-03-25T18:02:24.608 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:24.608 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:24.608 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:24.608 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:24.608 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:24.608 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:24.608 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:24.608 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:24.608 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:24.608 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:24.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1316927535 0 0) 0x7f0c6c138e40 con 0x7f0c6c1430c0 2024-03-25T18:02:24.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 >> v1:172.21.15.169:6789/0 conn(0x7f0c6c13f710 legacy=0x7f0c6c1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:24.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 >> v1:172.21.15.18:6789/0 conn(0x7f0c6c0a98f0 legacy=0x7f0c6c138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:24.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c68ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:02:24.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c68ff9640 1 monclient: found mon.c 2024-03-25T18:02:24.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c68ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:02:24.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c68ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:02:24.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c1d8340 con 0x7f0c6c1430c0 2024-03-25T18:02:24.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c68ff9640 10 monclient: _finish_auth 0 2024-03-25T18:02:24.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c68ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:02:24.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:02:24.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c723c4640 5 monclient: authenticate success, global_id 4508 2024-03-25T18:02:24.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c723c4640 10 monclient: _renew_subs 2024-03-25T18:02:24.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c723c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:02:24.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c723c4640 1 -- 172.21.15.18:0/1614327087 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6c134820 con 0x7f0c6c1430c0 2024-03-25T18:02:24.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c723c4640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:02:24.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c723c4640 10 monclient: _renew_subs 2024-03-25T18:02:24.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c723c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:02:24.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c723c4640 1 -- 172.21.15.18:0/1614327087 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6c1380c0 con 0x7f0c6c1430c0 2024-03-25T18:02:24.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.607+0000 7f0c723c4640 1 librados: init done 2024-03-25T18:02:24.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.608+0000 7f0c723c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:24.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.608+0000 7f0c723c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:24.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.608+0000 7f0c723c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:02:24.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.608+0000 7f0c723c4640 1 -- 172.21.15.18:0/1614327087 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c6c0627b0 con 0x7f0c6c1430c0 2024-03-25T18:02:24.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.913+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c64001fb0 con 0x7f0c6c1430c0 2024-03-25T18:02:24.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.913+0000 7f0c68ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:02:24.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.913+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0c64003b10 con 0x7f0c6c1430c0 2024-03-25T18:02:24.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.913+0000 7f0c68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:24.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.913+0000 7f0c68ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:02:24.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.913+0000 7f0c68ff9640 10 monclient: dump: 2024-03-25T18:02:24.915 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:24.915 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:24.915 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:24.915 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:24.915 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:24.915 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:24.915 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:24.915 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:24.916 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:24.916 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:24.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.913+0000 7f0c6a7fc640 4 set_mon_vals no callback set 2024-03-25T18:02:24.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.914+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f0c64003d30 con 0x7f0c6c1430c0 2024-03-25T18:02:24.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.915+0000 7f0c6bfff640 10 monclient: get_auth_request con 0x7f0c5806df70 auth_method 0 2024-03-25T18:02:24.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.915+0000 7f0c68ff9640 10 client.4508.objecter ms_handle_connect 0x7f0c5806df70 2024-03-25T18:02:24.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.920+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f0c64053780 con 0x7f0c6c1430c0 2024-03-25T18:02:24.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.920+0000 7f0c68ff9640 10 client.4508.objecter ms_dispatch 0x7f0c6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:02:24.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.920+0000 7f0c68ff9640 3 client.4508.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:02:24.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.920+0000 7f0c68ff9640 3 client.4508.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:02:24.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.920+0000 7f0c68ff9640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-25T18:02:24.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.920+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0c6404f290 con 0x7f0c6c1430c0 2024-03-25T18:02:24.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.920+0000 7f0c68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:24.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:24.920+0000 7f0c68ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:02:25.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.186+0000 7f0c723c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:25.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.186+0000 7f0c723c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:25.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.186+0000 7f0c723c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:02:25.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.186+0000 7f0c723c4640 1 -- 172.21.15.18:0/1614327087 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c6c14e880 con 0x7f0c6c1430c0 2024-03-25T18:02:25.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.319+0000 7f0c68ff9640 1 -- 172.21.15.18:0/1614327087 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f0c640511c0 con 0x7f0c6c1430c0 2024-03-25T18:02:25.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.319+0000 7f0c68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:25.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.319+0000 7f0c68ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:02:25.321 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:02:25.321 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:02:25.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.322+0000 7f0c723c4640 10 librados: watch_flush enter 2024-03-25T18:02:25.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.322+0000 7f0c723c4640 10 librados: watch_flush exit 2024-03-25T18:02:25.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.322+0000 7f0c723c4640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-25T18:02:25.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.322+0000 7f0c723c4640 10 client.4508.objecter successfully canceled tick 2024-03-25T18:02:25.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.322+0000 7f0c723c4640 1 -- 172.21.15.18:0/1614327087 >> v1:172.21.15.18:6800/3721244451 conn(0x7f0c5806df70 legacy=0x7f0c58070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:25.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.322+0000 7f0c723c4640 10 monclient: shutdown 2024-03-25T18:02:25.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.322+0000 7f0c723c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:02:25.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.322+0000 7f0c723c4640 1 -- 172.21.15.18:0/1614327087 >> v1:172.21.15.18:6790/0 conn(0x7f0c6c1430c0 legacy=0x7f0c6c134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:25.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.322+0000 7f0c723c4640 1 -- 172.21.15.18:0/1614327087 shutdown_connections 2024-03-25T18:02:25.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.322+0000 7f0c723c4640 1 -- 172.21.15.18:0/1614327087 >> 172.21.15.18:0/1614327087 conn(0x7f0c6c0b7ea0 msgr2=0x7f0c6c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:02:25.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.322+0000 7f0c723c4640 1 -- 172.21.15.18:0/1614327087 shutdown_connections 2024-03-25T18:02:25.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.322+0000 7f0c723c4640 1 -- 172.21.15.18:0/1614327087 wait complete. 2024-03-25T18:02:25.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:25.322+0000 7f0c723c4640 1 librados: shutdown 2024-03-25T18:02:25.335 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:02:32.140 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:02:32.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.254+0000 7fb9d0e9a640 10 monclient: get_monmap_and_config 2024-03-25T18:02:32.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.254+0000 7fb9d0e9a640 10 monclient: build_initial_monmap 2024-03-25T18:02:32.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.254+0000 7fb9d0e9a640 10 monclient: monmap: 2024-03-25T18:02:32.256 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:02:32.256 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:32.257 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:02:32.256281+0000 2024-03-25T18:02:32.257 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:02:32.256281+0000 2024-03-25T18:02:32.257 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:02:32.257 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:32.257 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:02:32.257 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:02:32.257 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:02:32.257 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:32.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.255+0000 7fb9d0e9a640 1 Processor -- start 2024-03-25T18:02:32.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.255+0000 7fb9d0e9a640 1 -- start start 2024-03-25T18:02:32.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.255+0000 7fb9d0e9a640 10 monclient: init 2024-03-25T18:02:32.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.256+0000 7fb9d0e9a640 10 monclient: _reopen_session rank -1 2024-03-25T18:02:32.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.256+0000 7fb9d0e9a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:02:32.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.256+0000 7fb9d0e9a640 10 monclient(hunting): picked mon.noname-b con 0x7fb9cc0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:02:32.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.256+0000 7fb9d0e9a640 10 monclient(hunting): picked mon.noname-a con 0x7fb9cc13f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:02:32.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.256+0000 7fb9d0e9a640 10 monclient(hunting): picked mon.noname-c con 0x7fb9cc1430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:02:32.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.256+0000 7fb9d0e9a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc142520 con 0x7fb9cc13f710 2024-03-25T18:02:32.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.256+0000 7fb9d0e9a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc0b28a0 con 0x7fb9cc1430c0 2024-03-25T18:02:32.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.256+0000 7fb9d0e9a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc0b2a30 con 0x7fb9cc0a98f0 2024-03-25T18:02:32.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.256+0000 7fb9d0e9a640 10 monclient(hunting): _renew_subs 2024-03-25T18:02:32.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.256+0000 7fb9d0e9a640 10 monclient(hunting): authenticate will time out at 7280.835303s 2024-03-25T18:02:32.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.267+0000 7fb9cad9d640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fb9cc1430c0 0x7fb9cc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:34718/0 (socket says 172.21.15.18:34718) 2024-03-25T18:02:32.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.267+0000 7fb9cad9d640 1 -- 172.21.15.18:0/3416759520 learned_addr learned my addr 172.21.15.18:0/3416759520 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:02:32.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.275+0000 7fb9c9d9b640 1 -- 172.21.15.18:0/3416759520 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364702980 0 0) 0x7fb9cc0b28a0 con 0x7fb9cc1430c0 2024-03-25T18:02:32.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.275+0000 7fb9c9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:02:32.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.275+0000 7fb9c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:32.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.275+0000 7fb9c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:32.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.275+0000 7fb9c9d9b640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:02:32.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.275+0000 7fb9c9d9b640 1 -- 172.21.15.18:0/3416759520 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9b8003520 con 0x7fb9cc1430c0 2024-03-25T18:02:32.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.286+0000 7fb9c9d9b640 1 -- 172.21.15.18:0/3416759520 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870682053 0 0) 0x7fb9cc0b2a30 con 0x7fb9cc0a98f0 2024-03-25T18:02:32.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.286+0000 7fb9c9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:02:32.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.286+0000 7fb9c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:32.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.286+0000 7fb9c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:32.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.286+0000 7fb9c9d9b640 10 monclient(hunting): my global_id is 4511 2024-03-25T18:02:32.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.286+0000 7fb9c9d9b640 1 -- 172.21.15.18:0/3416759520 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc0b28a0 con 0x7fb9cc0a98f0 2024-03-25T18:02:32.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.287+0000 7fb9c9d9b640 1 -- 172.21.15.18:0/3416759520 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206691845 0 0) 0x7fb9cc142520 con 0x7fb9cc13f710 2024-03-25T18:02:32.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.287+0000 7fb9c9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:02:32.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.287+0000 7fb9c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:32.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.287+0000 7fb9c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:32.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.287+0000 7fb9c9d9b640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:02:32.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.287+0000 7fb9c9d9b640 1 -- 172.21.15.18:0/3416759520 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc0b2a30 con 0x7fb9cc13f710 2024-03-25T18:02:32.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.289+0000 7fb9c9d9b640 1 -- 172.21.15.18:0/3416759520 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1006469988 0 0) 0x7fb9b8003520 con 0x7fb9cc1430c0 2024-03-25T18:02:32.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.289+0000 7fb9c9d9b640 1 -- 172.21.15.18:0/3416759520 >> v1:172.21.15.18:6790/0 conn(0x7fb9cc0a98f0 legacy=0x7fb9cc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:32.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.289+0000 7fb9c9d9b640 1 -- 172.21.15.18:0/3416759520 >> v1:172.21.15.18:6789/0 conn(0x7fb9cc13f710 legacy=0x7fb9cc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:32.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.289+0000 7fb9c9d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:02:32.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.289+0000 7fb9c9d9b640 1 monclient: found mon.noname-c 2024-03-25T18:02:32.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.289+0000 7fb9c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:02:32.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.289+0000 7fb9c9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:02:32.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.289+0000 7fb9c9d9b640 1 -- 172.21.15.18:0/3416759520 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc0b44c0 con 0x7fb9cc1430c0 2024-03-25T18:02:32.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.289+0000 7fb9c9d9b640 10 monclient: _finish_auth 0 2024-03-25T18:02:32.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.289+0000 7fb9c9d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:02:32.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.289+0000 7fb9c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:02:32.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.289+0000 7fb9c9d9b640 1 -- 172.21.15.18:0/3416759520 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fb9c4001a10 con 0x7fb9cc1430c0 2024-03-25T18:02:32.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.289+0000 7fb9c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:32.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.289+0000 7fb9c9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:02:32.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.289+0000 7fb9c9d9b640 10 monclient: dump: 2024-03-25T18:02:32.291 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:32.291 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:32.291 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:32.291 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:32.292 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:32.292 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:32.292 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:32.292 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:32.292 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:32.292 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:32.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.289+0000 7fb9d0e9a640 5 monclient: authenticate success, global_id 4492 2024-03-25T18:02:32.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.289+0000 7fb9d0e9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:02:32.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.303+0000 7fb9c9d9b640 1 -- 172.21.15.18:0/3416759520 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9c4001b70 con 0x7fb9cc1430c0 2024-03-25T18:02:32.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.303+0000 7fb9c9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:02:32.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.303+0000 7fb9c9d9b640 1 -- 172.21.15.18:0/3416759520 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fb9c4001f80 con 0x7fb9cc1430c0 2024-03-25T18:02:32.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.303+0000 7fb9c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:32.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.303+0000 7fb9c9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:02:32.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.303+0000 7fb9c9d9b640 10 monclient: dump: 2024-03-25T18:02:32.304 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:32.304 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:32.304 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:32.304 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:32.304 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:32.305 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:32.305 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:32.305 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:32.305 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:32.305 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:32.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.303+0000 7fb9d0e9a640 10 monclient: get_monmap_and_config success 2024-03-25T18:02:32.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.303+0000 7fb9d0e9a640 4 set_mon_vals no callback set 2024-03-25T18:02:32.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.303+0000 7fb9d0e9a640 10 monclient: shutdown 2024-03-25T18:02:32.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.303+0000 7fb9d0e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:02:32.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.303+0000 7fb9d0e9a640 1 -- 172.21.15.18:0/3416759520 >> v1:172.21.15.169:6789/0 conn(0x7fb9cc1430c0 legacy=0x7fb9cc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:32.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.303+0000 7fb9d0e9a640 1 -- 172.21.15.18:0/3416759520 shutdown_connections 2024-03-25T18:02:32.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.304+0000 7fb9d0e9a640 1 -- 172.21.15.18:0/3416759520 >> 172.21.15.18:0/3416759520 conn(0x7fb9cc0b7ea0 msgr2=0x7fb9cc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:02:32.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.304+0000 7fb9d0e9a640 1 -- 172.21.15.18:0/3416759520 shutdown_connections 2024-03-25T18:02:32.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.304+0000 7fb9d0e9a640 1 -- 172.21.15.18:0/3416759520 wait complete. 2024-03-25T18:02:32.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.304+0000 7fb9d0e9a640 10 monclient: build_initial_monmap 2024-03-25T18:02:32.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.304+0000 7fb9d0e9a640 10 monclient: monmap: 2024-03-25T18:02:32.307 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:02:32.307 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:02:32.307 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:02:32.307 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:02:32.307 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:02:32.307 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:02:32.307 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:02:32.307 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:02:32.307 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:02:32.308 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:32.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.305+0000 7fb9d0e9a640 1 librados: starting msgr at 2024-03-25T18:02:32.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.305+0000 7fb9d0e9a640 1 librados: starting objecter 2024-03-25T18:02:32.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.305+0000 7fb9d0e9a640 1 Processor -- start 2024-03-25T18:02:32.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.305+0000 7fb9d0e9a640 1 -- start start 2024-03-25T18:02:32.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.305+0000 7fb9d0e9a640 1 librados: setting wanted keys 2024-03-25T18:02:32.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.305+0000 7fb9d0e9a640 1 librados: calling monclient init 2024-03-25T18:02:32.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.305+0000 7fb9d0e9a640 10 monclient: init 2024-03-25T18:02:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.306+0000 7fb9d0e9a640 10 monclient: _reopen_session rank -1 2024-03-25T18:02:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.306+0000 7fb9d0e9a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:02:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.306+0000 7fb9d0e9a640 10 monclient(hunting): picked mon.noname-a con 0x7fb9cc0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:02:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.306+0000 7fb9d0e9a640 10 monclient(hunting): picked mon.noname-b con 0x7fb9cc13f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:02:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.306+0000 7fb9d0e9a640 10 monclient(hunting): picked mon.noname-c con 0x7fb9cc1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:02:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.306+0000 7fb9d0e9a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc137190 con 0x7fb9cc0a98f0 2024-03-25T18:02:32.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.306+0000 7fb9d0e9a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc1d41e0 con 0x7fb9cc13f710 2024-03-25T18:02:32.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.306+0000 7fb9d0e9a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc1d4430 con 0x7fb9cc1430c0 2024-03-25T18:02:32.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.306+0000 7fb9d0e9a640 10 monclient(hunting): _renew_subs 2024-03-25T18:02:32.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.306+0000 7fb9d0e9a640 10 monclient(hunting): authenticate will time out at 7280.884945s 2024-03-25T18:02:32.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.316+0000 7fb9ca59c640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fb9cc0a98f0 0x7fb9cc1d4950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:48418/0 (socket says 172.21.15.18:48418) 2024-03-25T18:02:32.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.316+0000 7fb9ca59c640 1 -- 172.21.15.18:0/1467396674 learned_addr learned my addr 172.21.15.18:0/1467396674 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:02:32.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.325+0000 7fb9c37fe640 10 client.?.objecter ms_handle_connect 0x7fb9cc0a98f0 2024-03-25T18:02:32.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.325+0000 7fb9c37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:32.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.333+0000 7fb9c37fe640 10 client.?.objecter ms_handle_connect 0x7fb9cc1430c0 2024-03-25T18:02:32.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.333+0000 7fb9c37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:32.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.333+0000 7fb9c37fe640 10 client.?.objecter ms_handle_connect 0x7fb9cc13f710 2024-03-25T18:02:32.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.333+0000 7fb9c37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:32.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.341+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199084736 0 0) 0x7fb9cc1d4430 con 0x7fb9cc1430c0 2024-03-25T18:02:32.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.341+0000 7fb9c37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:02:32.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.341+0000 7fb9c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:32.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.341+0000 7fb9c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:32.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.341+0000 7fb9c37fe640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:02:32.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.341+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ac004eb0 con 0x7fb9cc1430c0 2024-03-25T18:02:32.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.343+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218621513 0 0) 0x7fb9cc137190 con 0x7fb9cc0a98f0 2024-03-25T18:02:32.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.343+0000 7fb9c37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:02:32.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.343+0000 7fb9c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:32.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.343+0000 7fb9c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:32.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.343+0000 7fb9c37fe640 10 monclient(hunting): my global_id is 4506 2024-03-25T18:02:32.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.343+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc1d4430 con 0x7fb9cc0a98f0 2024-03-25T18:02:32.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.344+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720254254 0 0) 0x7fb9cc1d41e0 con 0x7fb9cc13f710 2024-03-25T18:02:32.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.344+0000 7fb9c37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:02:32.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.344+0000 7fb9c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:32.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.344+0000 7fb9c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:32.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.344+0000 7fb9c37fe640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:02:32.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.344+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc137190 con 0x7fb9cc13f710 2024-03-25T18:02:32.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.359+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 297205533 0 0) 0x7fb9cc137190 con 0x7fb9cc13f710 2024-03-25T18:02:32.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.359+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9cc1d41e0 con 0x7fb9cc13f710 2024-03-25T18:02:32.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.359+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fb9b4001830 con 0x7fb9cc13f710 2024-03-25T18:02:32.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.359+0000 7fb9c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:32.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.359+0000 7fb9c37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:02:32.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.359+0000 7fb9c37fe640 10 monclient(hunting): dump: 2024-03-25T18:02:32.383 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:32.383 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:32.383 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:32.383 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:32.383 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:32.383 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:32.383 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:32.383 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:32.383 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:32.383 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:32.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.359+0000 7fb9c37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:02:32.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.362+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 791846674 0 0) 0x7fb9cc1d4430 con 0x7fb9cc0a98f0 2024-03-25T18:02:32.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.362+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9cc137190 con 0x7fb9cc0a98f0 2024-03-25T18:02:32.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.362+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fb9bc0019b0 con 0x7fb9cc0a98f0 2024-03-25T18:02:32.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.362+0000 7fb9c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:32.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.362+0000 7fb9c37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:02:32.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.362+0000 7fb9c37fe640 10 monclient(hunting): dump: 2024-03-25T18:02:32.387 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:32.387 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:32.387 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:32.387 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:32.387 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:32.388 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:32.388 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:32.388 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:32.388 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:32.388 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:32.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.372+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2105327276 0 0) 0x7fb9ac004eb0 con 0x7fb9cc1430c0 2024-03-25T18:02:32.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.372+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9cc1d4430 con 0x7fb9cc1430c0 2024-03-25T18:02:32.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.372+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fb9c4001e80 con 0x7fb9cc1430c0 2024-03-25T18:02:32.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.372+0000 7fb9c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:32.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.372+0000 7fb9c37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:02:32.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.372+0000 7fb9c37fe640 10 monclient(hunting): dump: 2024-03-25T18:02:32.388 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:32.388 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:32.389 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:32.389 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:32.389 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:32.389 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:32.389 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:32.389 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:32.389 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:32.389 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:32.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.372+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1522503946 0 0) 0x7fb9cc1d41e0 con 0x7fb9cc13f710 2024-03-25T18:02:32.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 >> v1:172.21.15.18:6790/0 conn(0x7fb9cc1430c0 legacy=0x7fb9cc130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:32.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 >> v1:172.21.15.18:6789/0 conn(0x7fb9cc0a98f0 legacy=0x7fb9cc1d4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:32.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9c37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:02:32.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9c37fe640 1 monclient: found mon.b 2024-03-25T18:02:32.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:02:32.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9c37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:32.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc13bf90 con 0x7fb9cc13f710 2024-03-25T18:02:32.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9c37fe640 10 monclient: _finish_auth 0 2024-03-25T18:02:32.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9c37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:02:32.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:02:32.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9d0e9a640 5 monclient: authenticate success, global_id 4495 2024-03-25T18:02:32.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9d0e9a640 10 monclient: _renew_subs 2024-03-25T18:02:32.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9d0e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:32.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9d0e9a640 1 -- 172.21.15.18:0/1467396674 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9cc1d5060 con 0x7fb9cc13f710 2024-03-25T18:02:32.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9d0e9a640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:02:32.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9d0e9a640 10 monclient: _renew_subs 2024-03-25T18:02:32.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9d0e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:32.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9d0e9a640 1 -- 172.21.15.18:0/1467396674 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9cc139f00 con 0x7fb9cc13f710 2024-03-25T18:02:32.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.373+0000 7fb9d0e9a640 1 librados: init done 2024-03-25T18:02:32.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.374+0000 7fb9d0e9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:32.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.374+0000 7fb9d0e9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:32.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.374+0000 7fb9d0e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:32.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.374+0000 7fb9d0e9a640 1 -- 172.21.15.18:0/1467396674 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9cc1d45c0 con 0x7fb9cc13f710 2024-03-25T18:02:32.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.672+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9b4001d40 con 0x7fb9cc13f710 2024-03-25T18:02:32.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.672+0000 7fb9c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:02:32.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.672+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fb9b40037f0 con 0x7fb9cc13f710 2024-03-25T18:02:32.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.672+0000 7fb9c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:32.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.672+0000 7fb9c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:02:32.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.672+0000 7fb9c37fe640 10 monclient: dump: 2024-03-25T18:02:32.674 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:32.674 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:32.674 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:32.674 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:32.674 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:32.674 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:32.674 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:32.675 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:32.675 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:32.675 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:32.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.672+0000 7fb9c959a640 4 set_mon_vals no callback set 2024-03-25T18:02:32.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.673+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fb9b4019880 con 0x7fb9cc13f710 2024-03-25T18:02:32.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.674+0000 7fb9ca59c640 10 monclient: get_auth_request con 0x7fb9ac06e0f0 auth_method 0 2024-03-25T18:02:32.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.674+0000 7fb9c37fe640 10 client.4495.objecter ms_handle_connect 0x7fb9ac06e0f0 2024-03-25T18:02:32.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.678+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fb9b4053560 con 0x7fb9cc13f710 2024-03-25T18:02:32.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.678+0000 7fb9c37fe640 10 client.4495.objecter ms_dispatch 0x7fb9cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:02:32.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.678+0000 7fb9c37fe640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:02:32.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.678+0000 7fb9c37fe640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:02:32.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.678+0000 7fb9c37fe640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-25T18:02:32.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.679+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9b404eea0 con 0x7fb9cc13f710 2024-03-25T18:02:32.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.679+0000 7fb9c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:32.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.679+0000 7fb9c37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:02:32.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.931+0000 7fb9d0e9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:32.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.931+0000 7fb9d0e9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:32.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.931+0000 7fb9d0e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:32.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.931+0000 7fb9d0e9a640 1 -- 172.21.15.18:0/1467396674 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9cc0627b0 con 0x7fb9cc13f710 2024-03-25T18:02:32.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.969+0000 7fb9c37fe640 1 -- 172.21.15.18:0/1467396674 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fb9b4050fa0 con 0x7fb9cc13f710 2024-03-25T18:02:32.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.969+0000 7fb9c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:32.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.969+0000 7fb9c37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:02:32.971 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:02:32.971 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:02:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.971+0000 7fb9d0e9a640 10 librados: watch_flush enter 2024-03-25T18:02:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.971+0000 7fb9d0e9a640 10 librados: watch_flush exit 2024-03-25T18:02:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.971+0000 7fb9d0e9a640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-25T18:02:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.971+0000 7fb9d0e9a640 10 client.4495.objecter successfully canceled tick 2024-03-25T18:02:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.972+0000 7fb9d0e9a640 1 -- 172.21.15.18:0/1467396674 >> v1:172.21.15.18:6800/3721244451 conn(0x7fb9ac06e0f0 legacy=0x7fb9ac0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.972+0000 7fb9d0e9a640 10 monclient: shutdown 2024-03-25T18:02:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.972+0000 7fb9d0e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:02:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.972+0000 7fb9d0e9a640 1 -- 172.21.15.18:0/1467396674 >> v1:172.21.15.169:6789/0 conn(0x7fb9cc13f710 legacy=0x7fb9cc1352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.972+0000 7fb9d0e9a640 1 -- 172.21.15.18:0/1467396674 shutdown_connections 2024-03-25T18:02:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.972+0000 7fb9d0e9a640 1 -- 172.21.15.18:0/1467396674 >> 172.21.15.18:0/1467396674 conn(0x7fb9cc0b7ea0 msgr2=0x7fb9cc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:02:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.972+0000 7fb9d0e9a640 1 -- 172.21.15.18:0/1467396674 shutdown_connections 2024-03-25T18:02:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.972+0000 7fb9d0e9a640 1 -- 172.21.15.18:0/1467396674 wait complete. 2024-03-25T18:02:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:32.972+0000 7fb9d0e9a640 1 librados: shutdown 2024-03-25T18:02:32.984 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:02:39.791 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:02:39.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.907+0000 7f4bf1ec1640 10 monclient: get_monmap_and_config 2024-03-25T18:02:39.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.907+0000 7f4bf1ec1640 10 monclient: build_initial_monmap 2024-03-25T18:02:39.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.907+0000 7f4bf1ec1640 10 monclient: monmap: 2024-03-25T18:02:39.907 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:02:39.907 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:39.908 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:02:39.907436+0000 2024-03-25T18:02:39.908 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:02:39.907436+0000 2024-03-25T18:02:39.908 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:02:39.908 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:39.908 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:02:39.908 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:02:39.908 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:02:39.908 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:39.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.907+0000 7f4bf1ec1640 1 Processor -- start 2024-03-25T18:02:39.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.907+0000 7f4bf1ec1640 1 -- start start 2024-03-25T18:02:39.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.907+0000 7f4bf1ec1640 10 monclient: init 2024-03-25T18:02:39.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.908+0000 7f4bf1ec1640 10 monclient: _reopen_session rank -1 2024-03-25T18:02:39.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.908+0000 7f4bf1ec1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:02:39.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.908+0000 7f4bf1ec1640 10 monclient(hunting): picked mon.noname-b con 0x7f4bec0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:02:39.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.908+0000 7f4bf1ec1640 10 monclient(hunting): picked mon.noname-a con 0x7f4bec13f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:02:39.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.908+0000 7f4bf1ec1640 10 monclient(hunting): picked mon.noname-c con 0x7f4bec1430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:02:39.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.908+0000 7f4bf1ec1640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec142520 con 0x7f4bec13f710 2024-03-25T18:02:39.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.908+0000 7f4bf1ec1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec0b28a0 con 0x7f4bec1430c0 2024-03-25T18:02:39.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.908+0000 7f4bf1ec1640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec0b2a30 con 0x7f4bec0a98f0 2024-03-25T18:02:39.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.908+0000 7f4bf1ec1640 10 monclient(hunting): _renew_subs 2024-03-25T18:02:39.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.908+0000 7f4bf1ec1640 10 monclient(hunting): authenticate will time out at 7288.486393s 2024-03-25T18:02:39.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.916+0000 7f4bebfff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f4bec1430c0 0x7f4bec14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:41714/0 (socket says 172.21.15.18:41714) 2024-03-25T18:02:39.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.916+0000 7f4bebfff640 1 -- 172.21.15.18:0/251308006 learned_addr learned my addr 172.21.15.18:0/251308006 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:02:39.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.933+0000 7f4bea7fc640 1 -- 172.21.15.18:0/251308006 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858966838 0 0) 0x7f4bec0b2a30 con 0x7f4bec0a98f0 2024-03-25T18:02:39.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.933+0000 7f4bea7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:02:39.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.933+0000 7f4bea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:39.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.933+0000 7f4bea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:39.933 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.933+0000 7f4bea7fc640 10 monclient(hunting): my global_id is 4517 2024-03-25T18:02:39.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.933+0000 7f4bea7fc640 1 -- 172.21.15.18:0/251308006 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bd4003520 con 0x7f4bec0a98f0 2024-03-25T18:02:39.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.934+0000 7f4bea7fc640 1 -- 172.21.15.18:0/251308006 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337022895 0 0) 0x7f4bec142520 con 0x7f4bec13f710 2024-03-25T18:02:39.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.934+0000 7f4bea7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:02:39.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.934+0000 7f4bea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:39.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.934+0000 7f4bea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:39.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.934+0000 7f4bea7fc640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:02:39.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.934+0000 7f4bea7fc640 1 -- 172.21.15.18:0/251308006 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bec0b2a30 con 0x7f4bec13f710 2024-03-25T18:02:39.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.937+0000 7f4bea7fc640 1 -- 172.21.15.18:0/251308006 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040923444 0 0) 0x7f4bec0b28a0 con 0x7f4bec1430c0 2024-03-25T18:02:39.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.937+0000 7f4bea7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:02:39.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.937+0000 7f4bea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:39.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.937+0000 7f4bea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:39.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.937+0000 7f4bea7fc640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:02:39.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.937+0000 7f4bea7fc640 1 -- 172.21.15.18:0/251308006 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bec142520 con 0x7f4bec1430c0 2024-03-25T18:02:39.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bea7fc640 1 -- 172.21.15.18:0/251308006 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1276609135 0 0) 0x7f4bec0b2a30 con 0x7f4bec13f710 2024-03-25T18:02:39.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bea7fc640 1 -- 172.21.15.18:0/251308006 >> v1:172.21.15.18:6790/0 conn(0x7f4bec0a98f0 legacy=0x7f4bec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:39.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bea7fc640 1 -- 172.21.15.18:0/251308006 >> v1:172.21.15.169:6789/0 conn(0x7f4bec1430c0 legacy=0x7f4bec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:39.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bea7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:02:39.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bea7fc640 1 monclient: found mon.noname-a 2024-03-25T18:02:39.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:02:39.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bea7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:02:39.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bea7fc640 1 -- 172.21.15.18:0/251308006 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bec0b44c0 con 0x7f4bec13f710 2024-03-25T18:02:39.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bea7fc640 10 monclient: _finish_auth 0 2024-03-25T18:02:39.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bea7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:02:39.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:02:39.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bea7fc640 1 -- 172.21.15.18:0/251308006 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f4be0001670 con 0x7f4bec13f710 2024-03-25T18:02:39.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bf1ec1640 5 monclient: authenticate success, global_id 4509 2024-03-25T18:02:39.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bf1ec1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:02:39.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:39.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:02:39.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bea7fc640 10 monclient: dump: 2024-03-25T18:02:39.951 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:39.951 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:39.951 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:39.951 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:39.951 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:39.951 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:39.951 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:39.952 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:39.952 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:39.952 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:39.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.949+0000 7f4bf1ec1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:02:39.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.973+0000 7f4bea7fc640 1 -- 172.21.15.18:0/251308006 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4be00017d0 con 0x7f4bec13f710 2024-03-25T18:02:39.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.973+0000 7f4bea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:02:39.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.973+0000 7f4bea7fc640 1 -- 172.21.15.18:0/251308006 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f4be0001be0 con 0x7f4bec13f710 2024-03-25T18:02:39.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.973+0000 7f4bf1ec1640 10 monclient: get_monmap_and_config success 2024-03-25T18:02:39.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.973+0000 7f4bf1ec1640 4 set_mon_vals no callback set 2024-03-25T18:02:39.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.973+0000 7f4bea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:39.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.973+0000 7f4bea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:02:39.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.973+0000 7f4bea7fc640 10 monclient: dump: 2024-03-25T18:02:39.974 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:39.974 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:39.974 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:39.974 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:39.974 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:39.974 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:39.974 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:39.974 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:39.974 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:39.975 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:39.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.974+0000 7f4bf1ec1640 10 monclient: shutdown 2024-03-25T18:02:39.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.974+0000 7f4bf1ec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:02:39.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.974+0000 7f4bf1ec1640 1 -- 172.21.15.18:0/251308006 >> v1:172.21.15.18:6789/0 conn(0x7f4bec13f710 legacy=0x7f4bec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:39.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.974+0000 7f4bf1ec1640 1 -- 172.21.15.18:0/251308006 shutdown_connections 2024-03-25T18:02:39.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.974+0000 7f4bf1ec1640 1 -- 172.21.15.18:0/251308006 >> 172.21.15.18:0/251308006 conn(0x7f4bec0b7ea0 msgr2=0x7f4bec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:02:39.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.974+0000 7f4bf1ec1640 1 -- 172.21.15.18:0/251308006 shutdown_connections 2024-03-25T18:02:39.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.974+0000 7f4bf1ec1640 1 -- 172.21.15.18:0/251308006 wait complete. 2024-03-25T18:02:39.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.975+0000 7f4bf1ec1640 10 monclient: build_initial_monmap 2024-03-25T18:02:39.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.975+0000 7f4bf1ec1640 10 monclient: monmap: 2024-03-25T18:02:39.977 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:02:39.978 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:02:39.978 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:02:39.978 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:02:39.978 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:02:39.978 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:02:39.978 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:02:39.978 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:02:39.978 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:02:39.978 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:39.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.975+0000 7f4bf1ec1640 1 librados: starting msgr at 2024-03-25T18:02:39.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.975+0000 7f4bf1ec1640 1 librados: starting objecter 2024-03-25T18:02:39.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.975+0000 7f4bf1ec1640 1 Processor -- start 2024-03-25T18:02:39.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.975+0000 7f4bf1ec1640 1 -- start start 2024-03-25T18:02:39.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.975+0000 7f4bf1ec1640 1 librados: setting wanted keys 2024-03-25T18:02:39.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.975+0000 7f4bf1ec1640 1 librados: calling monclient init 2024-03-25T18:02:39.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.975+0000 7f4bf1ec1640 10 monclient: init 2024-03-25T18:02:39.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.976+0000 7f4bf1ec1640 10 monclient: _reopen_session rank -1 2024-03-25T18:02:39.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.976+0000 7f4bf1ec1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:02:39.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.976+0000 7f4bf1ec1640 10 monclient(hunting): picked mon.noname-c con 0x7f4bec0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:02:39.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.976+0000 7f4bf1ec1640 10 monclient(hunting): picked mon.noname-b con 0x7f4bec13f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:02:39.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.976+0000 7f4bf1ec1640 10 monclient(hunting): picked mon.noname-a con 0x7f4bec1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:02:39.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.976+0000 7f4bf1ec1640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec136750 con 0x7f4bec1430c0 2024-03-25T18:02:39.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.976+0000 7f4bf1ec1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec1dac30 con 0x7f4bec13f710 2024-03-25T18:02:39.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.976+0000 7f4bf1ec1640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec1399c0 con 0x7f4bec0a98f0 2024-03-25T18:02:39.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.976+0000 7f4bf1ec1640 10 monclient(hunting): _renew_subs 2024-03-25T18:02:39.979 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.976+0000 7f4bf1ec1640 10 monclient(hunting): authenticate will time out at 7288.554478s 2024-03-25T18:02:39.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.986+0000 7f4beaffd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f4bec13f710 0x7f4bec1db150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:41734/0 (socket says 172.21.15.18:41734) 2024-03-25T18:02:39.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.986+0000 7f4beaffd640 1 -- 172.21.15.18:0/141293125 learned_addr learned my addr 172.21.15.18:0/141293125 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:02:39.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.987+0000 7f4bd3fff640 10 client.?.objecter ms_handle_connect 0x7f4bec13f710 2024-03-25T18:02:39.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.987+0000 7f4bd3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:39.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.992+0000 7f4bd3fff640 10 client.?.objecter ms_handle_connect 0x7f4bec0a98f0 2024-03-25T18:02:39.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.992+0000 7f4bd3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:39.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.995+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414024860 0 0) 0x7f4bec1dac30 con 0x7f4bec13f710 2024-03-25T18:02:39.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.995+0000 7f4bd3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:02:39.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.995+0000 7f4bd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:39.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.995+0000 7f4bd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:39.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.995+0000 7f4bd3fff640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:02:39.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:39.995+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bd8004eb0 con 0x7f4bec13f710 2024-03-25T18:02:40.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.002+0000 7f4bd3fff640 10 client.?.objecter ms_handle_connect 0x7f4bec1430c0 2024-03-25T18:02:40.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.002+0000 7f4bd3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:40.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.008+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247176930 0 0) 0x7f4bec136750 con 0x7f4bec1430c0 2024-03-25T18:02:40.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.008+0000 7f4bd3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:02:40.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.008+0000 7f4bd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:40.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.008+0000 7f4bd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:40.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.008+0000 7f4bd3fff640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:02:40.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.008+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bec1dac30 con 0x7f4bec1430c0 2024-03-25T18:02:40.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.008+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2184985182 0 0) 0x7f4bd8004eb0 con 0x7f4bec13f710 2024-03-25T18:02:40.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.008+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bec136750 con 0x7f4bec13f710 2024-03-25T18:02:40.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.008+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f4be0001670 con 0x7f4bec13f710 2024-03-25T18:02:40.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.008+0000 7f4bd3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:40.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.008+0000 7f4bd3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:02:40.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.008+0000 7f4bd3fff640 10 monclient(hunting): dump: 2024-03-25T18:02:40.009 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:40.009 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:40.009 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:40.010 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:40.010 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:40.010 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:40.010 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:40.010 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:40.010 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:40.010 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:40.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.008+0000 7f4bd3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:02:40.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.015+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664907011 0 0) 0x7f4bec1399c0 con 0x7f4bec0a98f0 2024-03-25T18:02:40.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.015+0000 7f4bd3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:02:40.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.015+0000 7f4bd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:40.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.015+0000 7f4bd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:40.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.015+0000 7f4bd3fff640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:02:40.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.015+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bd8004eb0 con 0x7f4bec0a98f0 2024-03-25T18:02:40.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3259173989 0 0) 0x7f4bec136750 con 0x7f4bec13f710 2024-03-25T18:02:40.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 >> v1:172.21.15.18:6790/0 conn(0x7f4bec0a98f0 legacy=0x7f4bec1386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:40.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 >> v1:172.21.15.18:6789/0 conn(0x7f4bec1430c0 legacy=0x7f4bec1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:40.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bd3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:02:40.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bd3fff640 1 monclient: found mon.b 2024-03-25T18:02:40.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:02:40.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bd3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:40.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bec1db860 con 0x7f4bec13f710 2024-03-25T18:02:40.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bd3fff640 10 monclient: _finish_auth 0 2024-03-25T18:02:40.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bd3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:02:40.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:02:40.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bf1ec1640 5 monclient: authenticate success, global_id 4501 2024-03-25T18:02:40.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bf1ec1640 10 monclient: _renew_subs 2024-03-25T18:02:40.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bf1ec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:40.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bf1ec1640 1 -- 172.21.15.18:0/141293125 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bec1d41e0 con 0x7f4bec13f710 2024-03-25T18:02:40.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bf1ec1640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:02:40.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bf1ec1640 10 monclient: _renew_subs 2024-03-25T18:02:40.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bf1ec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:40.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bf1ec1640 1 -- 172.21.15.18:0/141293125 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bec1d5150 con 0x7f4bec13f710 2024-03-25T18:02:40.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.021+0000 7f4bf1ec1640 1 librados: init done 2024-03-25T18:02:40.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.022+0000 7f4bf1ec1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:40.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.022+0000 7f4bf1ec1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:40.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.022+0000 7f4bf1ec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:40.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.022+0000 7f4bf1ec1640 1 -- 172.21.15.18:0/141293125 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bec1daa90 con 0x7f4bec13f710 2024-03-25T18:02:40.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.311+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4be00038a0 con 0x7f4bec13f710 2024-03-25T18:02:40.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.311+0000 7f4bd3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:02:40.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.311+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f4be0003b10 con 0x7f4bec13f710 2024-03-25T18:02:40.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.311+0000 7f4bd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:40.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.311+0000 7f4bd3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:02:40.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.311+0000 7f4bd3fff640 10 monclient: dump: 2024-03-25T18:02:40.312 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:40.312 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:40.312 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:40.312 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:40.312 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:40.312 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:40.313 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:40.313 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:40.313 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:40.313 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:40.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.311+0000 7f4be9ffb640 4 set_mon_vals no callback set 2024-03-25T18:02:40.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.312+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f4be0003cf0 con 0x7f4bec13f710 2024-03-25T18:02:40.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.313+0000 7f4beb7fe640 10 monclient: get_auth_request con 0x7f4bd806de00 auth_method 0 2024-03-25T18:02:40.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.313+0000 7f4bd3fff640 10 client.4501.objecter ms_handle_connect 0x7f4bd806de00 2024-03-25T18:02:40.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.317+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f4be0054810 con 0x7f4bec13f710 2024-03-25T18:02:40.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.317+0000 7f4bd3fff640 10 client.4501.objecter ms_dispatch 0x7f4bec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:02:40.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.317+0000 7f4bd3fff640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:02:40.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.317+0000 7f4bd3fff640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:02:40.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.317+0000 7f4bd3fff640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-25T18:02:40.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.317+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4be0050320 con 0x7f4bec13f710 2024-03-25T18:02:40.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.317+0000 7f4bd3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:40.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.317+0000 7f4bd3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:02:40.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.577+0000 7f4bf1ec1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:40.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.577+0000 7f4bf1ec1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:40.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.577+0000 7f4bf1ec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:40.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.577+0000 7f4bf1ec1640 1 -- 172.21.15.18:0/141293125 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bec1dadc0 con 0x7f4bec13f710 2024-03-25T18:02:40.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.608+0000 7f4bd3fff640 1 -- 172.21.15.18:0/141293125 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f4be0052250 con 0x7f4bec13f710 2024-03-25T18:02:40.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.608+0000 7f4bd3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:40.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.608+0000 7f4bd3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:02:40.609 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:02:40.609 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:02:40.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.611+0000 7f4bf1ec1640 10 librados: watch_flush enter 2024-03-25T18:02:40.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.611+0000 7f4bf1ec1640 10 librados: watch_flush exit 2024-03-25T18:02:40.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.611+0000 7f4bf1ec1640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-25T18:02:40.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.611+0000 7f4bf1ec1640 10 client.4501.objecter successfully canceled tick 2024-03-25T18:02:40.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.611+0000 7f4bf1ec1640 1 -- 172.21.15.18:0/141293125 >> v1:172.21.15.18:6800/3721244451 conn(0x7f4bd806de00 legacy=0x7f4bd80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:40.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.611+0000 7f4bf1ec1640 10 monclient: shutdown 2024-03-25T18:02:40.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.611+0000 7f4bf1ec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:02:40.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.611+0000 7f4bf1ec1640 1 -- 172.21.15.18:0/141293125 >> v1:172.21.15.169:6789/0 conn(0x7f4bec13f710 legacy=0x7f4bec1db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:40.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.611+0000 7f4bf1ec1640 1 -- 172.21.15.18:0/141293125 shutdown_connections 2024-03-25T18:02:40.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.611+0000 7f4bf1ec1640 1 -- 172.21.15.18:0/141293125 >> 172.21.15.18:0/141293125 conn(0x7f4bec0b7ea0 msgr2=0x7f4bec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:02:40.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.611+0000 7f4bf1ec1640 1 -- 172.21.15.18:0/141293125 shutdown_connections 2024-03-25T18:02:40.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.611+0000 7f4bf1ec1640 1 -- 172.21.15.18:0/141293125 wait complete. 2024-03-25T18:02:40.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:40.611+0000 7f4bf1ec1640 1 librados: shutdown 2024-03-25T18:02:40.622 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:02:43.170 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:06:40.963 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:43.174 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:06:40.967 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:43.209 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:06:41.001 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:43.224 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:40.973 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:43.231 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:40.979 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:43.252 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:06:41.001 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:43.262 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:06:41.054 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:43.265 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:06:41.058 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:43.288 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:06:41.080 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:02:47.428 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:02:47.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.540+0000 7f3c0300d640 10 monclient: get_monmap_and_config 2024-03-25T18:02:47.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.540+0000 7f3c0300d640 10 monclient: build_initial_monmap 2024-03-25T18:02:47.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.540+0000 7f3c0300d640 10 monclient: monmap: 2024-03-25T18:02:47.540 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:02:47.540 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:47.540 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:02:47.540570+0000 2024-03-25T18:02:47.540 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:02:47.540570+0000 2024-03-25T18:02:47.540 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:02:47.540 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:47.541 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:02:47.541 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:02:47.541 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:02:47.541 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:47.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.541+0000 7f3c0300d640 1 Processor -- start 2024-03-25T18:02:47.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.541+0000 7f3c0300d640 1 -- start start 2024-03-25T18:02:47.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.541+0000 7f3c0300d640 10 monclient: init 2024-03-25T18:02:47.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.542+0000 7f3c0300d640 10 monclient: _reopen_session rank -1 2024-03-25T18:02:47.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.542+0000 7f3c0300d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:02:47.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.542+0000 7f3c0300d640 10 monclient(hunting): picked mon.noname-b con 0x7f3bfc0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:02:47.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.542+0000 7f3c0300d640 10 monclient(hunting): picked mon.noname-a con 0x7f3bfc13f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:02:47.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.542+0000 7f3c0300d640 10 monclient(hunting): picked mon.noname-c con 0x7f3bfc1430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:02:47.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.542+0000 7f3c0300d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc142520 con 0x7f3bfc13f710 2024-03-25T18:02:47.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.542+0000 7f3c0300d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc0b28a0 con 0x7f3bfc1430c0 2024-03-25T18:02:47.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.542+0000 7f3c0300d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc0b2a30 con 0x7f3bfc0a98f0 2024-03-25T18:02:47.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.542+0000 7f3c0300d640 10 monclient(hunting): _renew_subs 2024-03-25T18:02:47.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.542+0000 7f3c0300d640 10 monclient(hunting): authenticate will time out at 7296.119997s 2024-03-25T18:02:47.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.550+0000 7f3c00da9640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f3bfc0a98f0 0x7f3bfc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:57772/0 (socket says 172.21.15.18:57772) 2024-03-25T18:02:47.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.550+0000 7f3c00da9640 1 -- 172.21.15.18:0/2155479974 learned_addr learned my addr 172.21.15.18:0/2155479974 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:02:47.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.561+0000 7f3bfb7fe640 1 -- 172.21.15.18:0/2155479974 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226318350 0 0) 0x7f3bfc0b2a30 con 0x7f3bfc0a98f0 2024-03-25T18:02:47.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.561+0000 7f3bfb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:02:47.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.561+0000 7f3bfb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:47.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.561+0000 7f3bfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:47.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.561+0000 7f3bfb7fe640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:02:47.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.561+0000 7f3bfb7fe640 1 -- 172.21.15.18:0/2155479974 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be4003520 con 0x7f3bfc0a98f0 2024-03-25T18:02:47.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.566+0000 7f3bfb7fe640 1 -- 172.21.15.18:0/2155479974 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225735577 0 0) 0x7f3bfc0b28a0 con 0x7f3bfc1430c0 2024-03-25T18:02:47.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.566+0000 7f3bfb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:02:47.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.566+0000 7f3bfb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:47.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.566+0000 7f3bfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:47.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.566+0000 7f3bfb7fe640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:02:47.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.566+0000 7f3bfb7fe640 1 -- 172.21.15.18:0/2155479974 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bfc0b2a30 con 0x7f3bfc1430c0 2024-03-25T18:02:47.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.574+0000 7f3bfb7fe640 1 -- 172.21.15.18:0/2155479974 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182888867 0 0) 0x7f3bfc142520 con 0x7f3bfc13f710 2024-03-25T18:02:47.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.574+0000 7f3bfb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:02:47.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.574+0000 7f3bfb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:47.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.574+0000 7f3bfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:47.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.574+0000 7f3bfb7fe640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:02:47.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.574+0000 7f3bfb7fe640 1 -- 172.21.15.18:0/2155479974 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bfc0b28a0 con 0x7f3bfc13f710 2024-03-25T18:02:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.579+0000 7f3bfb7fe640 1 -- 172.21.15.18:0/2155479974 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1234761771 0 0) 0x7f3be4003520 con 0x7f3bfc0a98f0 2024-03-25T18:02:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.579+0000 7f3bfb7fe640 1 -- 172.21.15.18:0/2155479974 >> v1:172.21.15.169:6789/0 conn(0x7f3bfc1430c0 legacy=0x7f3bfc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.579+0000 7f3bfb7fe640 1 -- 172.21.15.18:0/2155479974 >> v1:172.21.15.18:6789/0 conn(0x7f3bfc13f710 legacy=0x7f3bfc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.579+0000 7f3bfb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:02:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.579+0000 7f3bfb7fe640 1 monclient: found mon.noname-b 2024-03-25T18:02:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.579+0000 7f3bfb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:02:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.579+0000 7f3bfb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:02:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.579+0000 7f3bfb7fe640 1 -- 172.21.15.18:0/2155479974 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc0b44c0 con 0x7f3bfc0a98f0 2024-03-25T18:02:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.579+0000 7f3bfb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:02:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.579+0000 7f3bfb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:02:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.579+0000 7f3bfb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:02:47.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.579+0000 7f3bfb7fe640 1 -- 172.21.15.18:0/2155479974 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3bec001690 con 0x7f3bfc0a98f0 2024-03-25T18:02:47.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.579+0000 7f3bfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:47.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.579+0000 7f3bfb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:02:47.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.579+0000 7f3bfb7fe640 10 monclient: dump: 2024-03-25T18:02:47.581 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:47.581 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:47.581 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:47.581 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:47.581 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:47.581 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:47.581 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:47.581 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:47.581 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:47.581 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:47.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.579+0000 7f3c0300d640 5 monclient: authenticate success, global_id 4523 2024-03-25T18:02:47.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.579+0000 7f3c0300d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:02:47.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.585+0000 7f3bfb7fe640 1 -- 172.21.15.18:0/2155479974 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bec0017f0 con 0x7f3bfc0a98f0 2024-03-25T18:02:47.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.585+0000 7f3bfb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:02:47.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.585+0000 7f3bfb7fe640 1 -- 172.21.15.18:0/2155479974 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3bec001c00 con 0x7f3bfc0a98f0 2024-03-25T18:02:47.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.585+0000 7f3bfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:47.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.585+0000 7f3bfb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:02:47.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.585+0000 7f3bfb7fe640 10 monclient: dump: 2024-03-25T18:02:47.586 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:47.586 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:47.586 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:47.586 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:47.586 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:47.586 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:47.586 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:47.586 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:47.586 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:47.586 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:47.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.585+0000 7f3c0300d640 10 monclient: get_monmap_and_config success 2024-03-25T18:02:47.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.585+0000 7f3c0300d640 4 set_mon_vals no callback set 2024-03-25T18:02:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.586+0000 7f3c0300d640 10 monclient: shutdown 2024-03-25T18:02:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.586+0000 7f3c0300d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:02:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.586+0000 7f3c0300d640 1 -- 172.21.15.18:0/2155479974 >> v1:172.21.15.18:6790/0 conn(0x7f3bfc0a98f0 legacy=0x7f3bfc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.586+0000 7f3c0300d640 1 -- 172.21.15.18:0/2155479974 shutdown_connections 2024-03-25T18:02:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.586+0000 7f3c0300d640 1 -- 172.21.15.18:0/2155479974 >> 172.21.15.18:0/2155479974 conn(0x7f3bfc0b7ea0 msgr2=0x7f3bfc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:02:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.586+0000 7f3c0300d640 1 -- 172.21.15.18:0/2155479974 shutdown_connections 2024-03-25T18:02:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.586+0000 7f3c0300d640 1 -- 172.21.15.18:0/2155479974 wait complete. 2024-03-25T18:02:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.587+0000 7f3c0300d640 10 monclient: build_initial_monmap 2024-03-25T18:02:47.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.587+0000 7f3c0300d640 10 monclient: monmap: 2024-03-25T18:02:47.589 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:02:47.589 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:02:47.589 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:02:47.590 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:02:47.590 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:02:47.590 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:02:47.590 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:02:47.590 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:02:47.590 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:02:47.590 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:47.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.587+0000 7f3c0300d640 1 librados: starting msgr at 2024-03-25T18:02:47.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.587+0000 7f3c0300d640 1 librados: starting objecter 2024-03-25T18:02:47.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.587+0000 7f3c0300d640 1 Processor -- start 2024-03-25T18:02:47.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.587+0000 7f3c0300d640 1 -- start start 2024-03-25T18:02:47.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.587+0000 7f3c0300d640 1 librados: setting wanted keys 2024-03-25T18:02:47.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.587+0000 7f3c0300d640 1 librados: calling monclient init 2024-03-25T18:02:47.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.587+0000 7f3c0300d640 10 monclient: init 2024-03-25T18:02:47.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.588+0000 7f3c0300d640 10 monclient: _reopen_session rank -1 2024-03-25T18:02:47.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.588+0000 7f3c0300d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:02:47.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.588+0000 7f3c0300d640 10 monclient(hunting): picked mon.noname-a con 0x7f3bfc0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:02:47.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.588+0000 7f3c0300d640 10 monclient(hunting): picked mon.noname-c con 0x7f3bfc13f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:02:47.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.588+0000 7f3c0300d640 10 monclient(hunting): picked mon.noname-b con 0x7f3bfc1430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:02:47.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.588+0000 7f3c0300d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc137ab0 con 0x7f3bfc0a98f0 2024-03-25T18:02:47.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.588+0000 7f3c0300d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc138bd0 con 0x7f3bfc1430c0 2024-03-25T18:02:47.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.588+0000 7f3c0300d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc138e40 con 0x7f3bfc13f710 2024-03-25T18:02:47.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.588+0000 7f3c0300d640 10 monclient(hunting): _renew_subs 2024-03-25T18:02:47.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.588+0000 7f3c0300d640 10 monclient(hunting): authenticate will time out at 7296.166112s 2024-03-25T18:02:47.601 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.601+0000 7f3c015aa640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f3bfc1430c0 0x7f3bfc1e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:56436/0 (socket says 172.21.15.18:56436) 2024-03-25T18:02:47.601 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.601+0000 7f3c015aa640 1 -- 172.21.15.18:0/4187538544 learned_addr learned my addr 172.21.15.18:0/4187538544 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:02:47.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.604+0000 7f3bf97fa640 10 client.?.objecter ms_handle_connect 0x7f3bfc1430c0 2024-03-25T18:02:47.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.604+0000 7f3bf97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:47.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.606+0000 7f3bf97fa640 10 client.?.objecter ms_handle_connect 0x7f3bfc13f710 2024-03-25T18:02:47.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.606+0000 7f3bf97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:47.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.616+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138177698 0 0) 0x7f3bfc138e40 con 0x7f3bfc13f710 2024-03-25T18:02:47.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.616+0000 7f3bf97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:02:47.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.616+0000 7f3bf97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:47.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.616+0000 7f3bf97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:47.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.616+0000 7f3bf97fa640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:02:47.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.616+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be8004eb0 con 0x7f3bfc13f710 2024-03-25T18:02:47.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.616+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018781111 0 0) 0x7f3bfc138bd0 con 0x7f3bfc1430c0 2024-03-25T18:02:47.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.616+0000 7f3bf97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:02:47.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.616+0000 7f3bf97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:47.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.616+0000 7f3bf97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:47.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.616+0000 7f3bf97fa640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:02:47.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.616+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bfc138e40 con 0x7f3bfc1430c0 2024-03-25T18:02:47.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.618+0000 7f3bf97fa640 10 client.?.objecter ms_handle_connect 0x7f3bfc0a98f0 2024-03-25T18:02:47.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.618+0000 7f3bf97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:47.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.627+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921018922 0 0) 0x7f3bfc137ab0 con 0x7f3bfc0a98f0 2024-03-25T18:02:47.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.628+0000 7f3bf97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:02:47.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.628+0000 7f3bf97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:47.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.628+0000 7f3bf97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:47.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.628+0000 7f3bf97fa640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:02:47.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.628+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bfc138bd0 con 0x7f3bfc0a98f0 2024-03-25T18:02:47.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.630+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 647411391 0 0) 0x7f3bfc138e40 con 0x7f3bfc1430c0 2024-03-25T18:02:47.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.630+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bfc137ab0 con 0x7f3bfc1430c0 2024-03-25T18:02:47.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.630+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3bf4001c50 con 0x7f3bfc1430c0 2024-03-25T18:02:47.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.630+0000 7f3bf97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:47.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.630+0000 7f3bf97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:02:47.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.630+0000 7f3bf97fa640 10 monclient(hunting): dump: 2024-03-25T18:02:47.631 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:47.631 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:47.631 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:47.631 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:47.631 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:47.631 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:47.631 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:47.631 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:47.631 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:47.631 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:47.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.630+0000 7f3bf97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:02:47.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.651+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3309694984 0 0) 0x7f3be8004eb0 con 0x7f3bfc13f710 2024-03-25T18:02:47.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.651+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bfc138e40 con 0x7f3bfc13f710 2024-03-25T18:02:47.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.651+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3bf00018f0 con 0x7f3bfc13f710 2024-03-25T18:02:47.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.651+0000 7f3bf97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:47.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.651+0000 7f3bf97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:02:47.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.651+0000 7f3bf97fa640 10 monclient(hunting): dump: 2024-03-25T18:02:47.653 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:47.653 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:47.653 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:47.653 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:47.654 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:47.654 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:47.654 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:47.654 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:47.654 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:47.654 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:47.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.660+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4123005310 0 0) 0x7f3bfc138bd0 con 0x7f3bfc0a98f0 2024-03-25T18:02:47.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.660+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3be8004eb0 con 0x7f3bfc0a98f0 2024-03-25T18:02:47.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.660+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3bec001b00 con 0x7f3bfc0a98f0 2024-03-25T18:02:47.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.660+0000 7f3bf97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:47.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.660+0000 7f3bf97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:02:47.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.660+0000 7f3bf97fa640 10 monclient(hunting): dump: 2024-03-25T18:02:47.661 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:47.661 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:47.661 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:47.661 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:47.661 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:47.661 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:47.661 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:47.661 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:47.661 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:47.661 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:47.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.662+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 595033954 0 0) 0x7f3bfc137ab0 con 0x7f3bfc1430c0 2024-03-25T18:02:47.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.662+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 >> v1:172.21.15.18:6790/0 conn(0x7f3bfc13f710 legacy=0x7f3bfc1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:47.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 >> v1:172.21.15.18:6789/0 conn(0x7f3bfc0a98f0 legacy=0x7f3bfc137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:47.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3bf97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:02:47.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3bf97fa640 1 monclient: found mon.b 2024-03-25T18:02:47.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3bf97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:02:47.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3bf97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:47.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc139120 con 0x7f3bfc1430c0 2024-03-25T18:02:47.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3bf97fa640 10 monclient: _finish_auth 0 2024-03-25T18:02:47.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3bf97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:02:47.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3bf97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:02:47.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3c0300d640 5 monclient: authenticate success, global_id 4507 2024-03-25T18:02:47.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3c0300d640 10 monclient: _renew_subs 2024-03-25T18:02:47.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3c0300d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:47.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3c0300d640 1 -- 172.21.15.18:0/4187538544 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bfc139400 con 0x7f3bfc1430c0 2024-03-25T18:02:47.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3c0300d640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:02:47.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3c0300d640 10 monclient: _renew_subs 2024-03-25T18:02:47.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3c0300d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:47.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3c0300d640 1 -- 172.21.15.18:0/4187538544 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bfc0b7880 con 0x7f3bfc1430c0 2024-03-25T18:02:47.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.663+0000 7f3c0300d640 1 librados: init done 2024-03-25T18:02:47.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.664+0000 7f3c0300d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:47.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.664+0000 7f3c0300d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:47.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.664+0000 7f3c0300d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:47.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.664+0000 7f3c0300d640 1 -- 172.21.15.18:0/4187538544 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bfc137810 con 0x7f3bfc1430c0 2024-03-25T18:02:47.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.971+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bf4003990 con 0x7f3bfc1430c0 2024-03-25T18:02:47.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.971+0000 7f3bf97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:02:47.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.971+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3bf4003d20 con 0x7f3bfc1430c0 2024-03-25T18:02:47.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.971+0000 7f3bf97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:47.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.971+0000 7f3bf97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:02:47.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.971+0000 7f3bf97fa640 10 monclient: dump: 2024-03-25T18:02:47.972 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:47.972 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:47.972 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:47.972 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:47.972 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:47.972 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:47.972 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:47.972 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:47.972 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:47.972 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:47.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.971+0000 7f3bfaffd640 4 set_mon_vals no callback set 2024-03-25T18:02:47.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.972+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f3bf4003f00 con 0x7f3bfc1430c0 2024-03-25T18:02:47.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.973+0000 7f3c00da9640 10 monclient: get_auth_request con 0x7f3be806e0f0 auth_method 0 2024-03-25T18:02:47.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.973+0000 7f3bf97fa640 10 client.4507.objecter ms_handle_connect 0x7f3be806e0f0 2024-03-25T18:02:47.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.977+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f3bf4054b50 con 0x7f3bfc1430c0 2024-03-25T18:02:47.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.977+0000 7f3bf97fa640 10 client.4507.objecter ms_dispatch 0x7f3bfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:02:47.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.977+0000 7f3bf97fa640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:02:47.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.977+0000 7f3bf97fa640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:02:47.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.977+0000 7f3bf97fa640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-25T18:02:47.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.977+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3bf4050660 con 0x7f3bfc1430c0 2024-03-25T18:02:47.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.977+0000 7f3bf97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:47.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:47.977+0000 7f3bf97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:02:48.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.233+0000 7f3c0300d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:48.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.233+0000 7f3c0300d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:48.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.233+0000 7f3c0300d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:48.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.233+0000 7f3c0300d640 1 -- 172.21.15.18:0/4187538544 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bfc0627b0 con 0x7f3bfc1430c0 2024-03-25T18:02:48.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.270+0000 7f3bf97fa640 1 -- 172.21.15.18:0/4187538544 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f3bf4052590 con 0x7f3bfc1430c0 2024-03-25T18:02:48.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.270+0000 7f3bf97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:48.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.270+0000 7f3bf97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:02:48.271 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:02:48.271 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:02:48.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.273+0000 7f3c0300d640 10 librados: watch_flush enter 2024-03-25T18:02:48.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.273+0000 7f3c0300d640 10 librados: watch_flush exit 2024-03-25T18:02:48.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.273+0000 7f3c0300d640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-25T18:02:48.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.273+0000 7f3c0300d640 10 client.4507.objecter successfully canceled tick 2024-03-25T18:02:48.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.273+0000 7f3c0300d640 1 -- 172.21.15.18:0/4187538544 >> v1:172.21.15.18:6800/3721244451 conn(0x7f3be806e0f0 legacy=0x7f3be80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:48.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.273+0000 7f3c0300d640 10 monclient: shutdown 2024-03-25T18:02:48.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.273+0000 7f3c0300d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:02:48.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.273+0000 7f3c0300d640 1 -- 172.21.15.18:0/4187538544 >> v1:172.21.15.169:6789/0 conn(0x7f3bfc1430c0 legacy=0x7f3bfc1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:48.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.273+0000 7f3c0300d640 1 -- 172.21.15.18:0/4187538544 shutdown_connections 2024-03-25T18:02:48.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.273+0000 7f3c0300d640 1 -- 172.21.15.18:0/4187538544 >> 172.21.15.18:0/4187538544 conn(0x7f3bfc0b7ea0 msgr2=0x7f3bfc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:02:48.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.274+0000 7f3c0300d640 1 -- 172.21.15.18:0/4187538544 shutdown_connections 2024-03-25T18:02:48.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.274+0000 7f3c0300d640 1 -- 172.21.15.18:0/4187538544 wait complete. 2024-03-25T18:02:48.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:48.274+0000 7f3c0300d640 1 librados: shutdown 2024-03-25T18:02:48.285 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:02:55.092 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:02:55.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.186+0000 7fd68bdb6640 10 monclient: get_monmap_and_config 2024-03-25T18:02:55.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.186+0000 7fd68bdb6640 10 monclient: build_initial_monmap 2024-03-25T18:02:55.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.186+0000 7fd68bdb6640 10 monclient: monmap: 2024-03-25T18:02:55.187 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:02:55.187 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:55.187 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:02:55.186649+0000 2024-03-25T18:02:55.187 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:02:55.186649+0000 2024-03-25T18:02:55.187 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:02:55.187 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:55.187 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:02:55.187 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:02:55.187 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:02:55.187 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:55.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.186+0000 7fd68bdb6640 1 Processor -- start 2024-03-25T18:02:55.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.187+0000 7fd68bdb6640 1 -- start start 2024-03-25T18:02:55.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.187+0000 7fd68bdb6640 10 monclient: init 2024-03-25T18:02:55.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.187+0000 7fd68bdb6640 10 monclient: _reopen_session rank -1 2024-03-25T18:02:55.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.187+0000 7fd68bdb6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:02:55.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.187+0000 7fd68bdb6640 10 monclient(hunting): picked mon.noname-a con 0x7fd6840a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:02:55.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.187+0000 7fd68bdb6640 10 monclient(hunting): picked mon.noname-b con 0x7fd68413f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:02:55.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.187+0000 7fd68bdb6640 10 monclient(hunting): picked mon.noname-c con 0x7fd6841430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:02:55.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.187+0000 7fd68bdb6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd684142520 con 0x7fd6840a98f0 2024-03-25T18:02:55.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.187+0000 7fd68bdb6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6840b28a0 con 0x7fd6841430c0 2024-03-25T18:02:55.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.187+0000 7fd68bdb6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6840b2a30 con 0x7fd68413f710 2024-03-25T18:02:55.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.187+0000 7fd68bdb6640 10 monclient(hunting): _renew_subs 2024-03-25T18:02:55.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.187+0000 7fd68bdb6640 10 monclient(hunting): authenticate will time out at 7303.765491s 2024-03-25T18:02:55.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.202+0000 7fd689b52640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fd6840a98f0 0x7fd6840a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:34216/0 (socket says 172.21.15.18:34216) 2024-03-25T18:02:55.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.202+0000 7fd689b52640 1 -- 172.21.15.18:0/3074991477 learned_addr learned my addr 172.21.15.18:0/3074991477 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:02:55.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.210+0000 7fd688b50640 1 -- 172.21.15.18:0/3074991477 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709460470 0 0) 0x7fd684142520 con 0x7fd6840a98f0 2024-03-25T18:02:55.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.210+0000 7fd688b50640 10 monclient(hunting): _negotiate 2024-03-25T18:02:55.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.210+0000 7fd688b50640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:55.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.210+0000 7fd688b50640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:55.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.210+0000 7fd688b50640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:02:55.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.210+0000 7fd688b50640 1 -- 172.21.15.18:0/3074991477 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd670003520 con 0x7fd6840a98f0 2024-03-25T18:02:55.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd688b50640 1 -- 172.21.15.18:0/3074991477 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1560510602 0 0) 0x7fd670003520 con 0x7fd6840a98f0 2024-03-25T18:02:55.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd688b50640 1 -- 172.21.15.18:0/3074991477 >> v1:172.21.15.18:6790/0 conn(0x7fd68413f710 legacy=0x7fd6841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:55.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd688b50640 1 -- 172.21.15.18:0/3074991477 >> v1:172.21.15.169:6789/0 conn(0x7fd6841430c0 legacy=0x7fd68414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:02:55.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd688b50640 10 monclient: _finish_hunting 0 2024-03-25T18:02:55.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd688b50640 1 monclient: found mon.noname-a 2024-03-25T18:02:55.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd688b50640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:02:55.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd688b50640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:02:55.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd688b50640 1 -- 172.21.15.18:0/3074991477 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6840b44c0 con 0x7fd6840a98f0 2024-03-25T18:02:55.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd688b50640 10 monclient: _finish_auth 0 2024-03-25T18:02:55.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd688b50640 10 monclient: _check_auth_tickets 2024-03-25T18:02:55.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd688b50640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:02:55.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd688b50640 1 -- 172.21.15.18:0/3074991477 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd680001690 con 0x7fd6840a98f0 2024-03-25T18:02:55.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd68bdb6640 5 monclient: authenticate success, global_id 4521 2024-03-25T18:02:55.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd68bdb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:02:55.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd688b50640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:55.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd688b50640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:02:55.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd688b50640 10 monclient: dump: 2024-03-25T18:02:55.223 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:55.223 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:55.223 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:55.223 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:55.224 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:55.224 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:55.224 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:55.224 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:55.224 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:55.224 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:55.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.221+0000 7fd68bdb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:02:55.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.227+0000 7fd688b50640 1 -- 172.21.15.18:0/3074991477 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd680001920 con 0x7fd6840a98f0 2024-03-25T18:02:55.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.227+0000 7fd688b50640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:02:55.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.227+0000 7fd688b50640 1 -- 172.21.15.18:0/3074991477 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd680001d30 con 0x7fd6840a98f0 2024-03-25T18:02:55.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.227+0000 7fd68bdb6640 10 monclient: get_monmap_and_config success 2024-03-25T18:02:55.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.227+0000 7fd68bdb6640 4 set_mon_vals no callback set 2024-03-25T18:02:55.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.227+0000 7fd688b50640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:55.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.227+0000 7fd688b50640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:02:55.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.227+0000 7fd688b50640 10 monclient: dump: 2024-03-25T18:02:55.228 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:55.228 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:55.228 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:55.228 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:55.228 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:55.229 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:55.229 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:55.229 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:55.229 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:55.229 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:55.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.228+0000 7fd68bdb6640 10 monclient: shutdown 2024-03-25T18:02:55.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.228+0000 7fd68bdb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:02:55.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.228+0000 7fd68bdb6640 1 -- 172.21.15.18:0/3074991477 >> v1:172.21.15.18:6789/0 conn(0x7fd6840a98f0 legacy=0x7fd6840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:55.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.228+0000 7fd68bdb6640 1 -- 172.21.15.18:0/3074991477 shutdown_connections 2024-03-25T18:02:55.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.228+0000 7fd68bdb6640 1 -- 172.21.15.18:0/3074991477 >> 172.21.15.18:0/3074991477 conn(0x7fd6840b7ea0 msgr2=0x7fd6840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:02:55.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.228+0000 7fd68bdb6640 1 -- 172.21.15.18:0/3074991477 shutdown_connections 2024-03-25T18:02:55.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.228+0000 7fd68bdb6640 1 -- 172.21.15.18:0/3074991477 wait complete. 2024-03-25T18:02:55.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.229+0000 7fd68bdb6640 10 monclient: build_initial_monmap 2024-03-25T18:02:55.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.229+0000 7fd68bdb6640 10 monclient: monmap: 2024-03-25T18:02:55.232 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:02:55.232 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:02:55.232 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:02:55.232 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:02:55.232 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:02:55.232 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:02:55.232 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:02:55.232 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:02:55.232 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:02:55.232 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:55.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.229+0000 7fd68bdb6640 1 librados: starting msgr at 2024-03-25T18:02:55.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.229+0000 7fd68bdb6640 1 librados: starting objecter 2024-03-25T18:02:55.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.229+0000 7fd68bdb6640 1 Processor -- start 2024-03-25T18:02:55.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.229+0000 7fd68bdb6640 1 -- start start 2024-03-25T18:02:55.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.229+0000 7fd68bdb6640 1 librados: setting wanted keys 2024-03-25T18:02:55.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.229+0000 7fd68bdb6640 1 librados: calling monclient init 2024-03-25T18:02:55.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.229+0000 7fd68bdb6640 10 monclient: init 2024-03-25T18:02:55.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.230+0000 7fd68bdb6640 10 monclient: _reopen_session rank -1 2024-03-25T18:02:55.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.230+0000 7fd68bdb6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:02:55.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.230+0000 7fd68bdb6640 10 monclient(hunting): picked mon.noname-c con 0x7fd6840a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:02:55.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.230+0000 7fd68bdb6640 10 monclient(hunting): picked mon.noname-b con 0x7fd68413f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:02:55.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.230+0000 7fd68bdb6640 10 monclient(hunting): picked mon.noname-a con 0x7fd6841430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:02:55.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.230+0000 7fd68bdb6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68413cc20 con 0x7fd6841430c0 2024-03-25T18:02:55.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.230+0000 7fd68bdb6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6840b7530 con 0x7fd68413f710 2024-03-25T18:02:55.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.230+0000 7fd68bdb6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6841dc090 con 0x7fd6840a98f0 2024-03-25T18:02:55.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.230+0000 7fd68bdb6640 10 monclient(hunting): _renew_subs 2024-03-25T18:02:55.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.230+0000 7fd68bdb6640 10 monclient(hunting): authenticate will time out at 7303.807994s 2024-03-25T18:02:55.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.240+0000 7fd689351640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fd68413f710 0x7fd6841db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:50818/0 (socket says 172.21.15.18:50818) 2024-03-25T18:02:55.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.240+0000 7fd689351640 1 -- 172.21.15.18:0/3084492721 learned_addr learned my addr 172.21.15.18:0/3084492721 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:02:55.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.241+0000 7fd67a7fc640 10 client.?.objecter ms_handle_connect 0x7fd68413f710 2024-03-25T18:02:55.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.241+0000 7fd67a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.244+0000 7fd67a7fc640 10 client.?.objecter ms_handle_connect 0x7fd6840a98f0 2024-03-25T18:02:55.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.244+0000 7fd67a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:55.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.256+0000 7fd67a7fc640 10 client.?.objecter ms_handle_connect 0x7fd6841430c0 2024-03-25T18:02:55.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.256+0000 7fd67a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:02:55.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.256+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094040452 0 0) 0x7fd6840b7530 con 0x7fd68413f710 2024-03-25T18:02:55.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.256+0000 7fd67a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:02:55.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.256+0000 7fd67a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:55.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.256+0000 7fd67a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:55.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.256+0000 7fd67a7fc640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:02:55.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.256+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd668004eb0 con 0x7fd68413f710 2024-03-25T18:02:55.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.266+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3317751220 0 0) 0x7fd668004eb0 con 0x7fd68413f710 2024-03-25T18:02:55.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.266+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6840b7530 con 0x7fd68413f710 2024-03-25T18:02:55.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.266+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd6740019f0 con 0x7fd68413f710 2024-03-25T18:02:55.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.266+0000 7fd67a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:55.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.266+0000 7fd67a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:02:55.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.266+0000 7fd67a7fc640 10 monclient(hunting): dump: 2024-03-25T18:02:55.267 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:55.267 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:55.267 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:55.267 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:55.267 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:55.267 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:55.267 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:55.267 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:55.268 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:55.268 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:55.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.266+0000 7fd67a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:02:55.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.267+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012466192 0 0) 0x7fd67c001370 con 0x7fd6841430c0 2024-03-25T18:02:55.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.267+0000 7fd67a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:02:55.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.267+0000 7fd67a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:55.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.267+0000 7fd67a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:55.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.267+0000 7fd67a7fc640 10 monclient(hunting): my global_id is 4524 2024-03-25T18:02:55.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.267+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd668004eb0 con 0x7fd6841430c0 2024-03-25T18:02:55.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.274+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935766457 0 0) 0x7fd6841dc090 con 0x7fd6840a98f0 2024-03-25T18:02:55.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.274+0000 7fd67a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:02:55.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.274+0000 7fd67a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:02:55.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.274+0000 7fd67a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:02:55.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.274+0000 7fd67a7fc640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:02:55.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.274+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd67c001370 con 0x7fd6840a98f0 2024-03-25T18:02:55.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.281+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 74792855 0 0) 0x7fd668004eb0 con 0x7fd6841430c0 2024-03-25T18:02:55.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.281+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6841dc090 con 0x7fd6841430c0 2024-03-25T18:02:55.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.281+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd67c001a00 con 0x7fd6841430c0 2024-03-25T18:02:55.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.281+0000 7fd67a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:55.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.281+0000 7fd67a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:02:55.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.281+0000 7fd67a7fc640 10 monclient(hunting): dump: 2024-03-25T18:02:55.282 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:55.282 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:55.282 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:55.282 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:55.282 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:55.282 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:55.282 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:55.282 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:55.282 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:55.282 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:55.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1890158036 0 0) 0x7fd6840b7530 con 0x7fd68413f710 2024-03-25T18:02:55.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 >> v1:172.21.15.18:6790/0 conn(0x7fd6840a98f0 legacy=0x7fd684139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:55.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 >> v1:172.21.15.18:6789/0 conn(0x7fd6841430c0 legacy=0x7fd6841db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:55.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd67a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:02:55.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd67a7fc640 1 monclient: found mon.b 2024-03-25T18:02:55.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd67a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:02:55.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd67a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:55.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68413b7e0 con 0x7fd68413f710 2024-03-25T18:02:55.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd67a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:02:55.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd67a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:02:55.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd67a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:02:55.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd68bdb6640 5 monclient: authenticate success, global_id 4510 2024-03-25T18:02:55.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd68bdb6640 10 monclient: _renew_subs 2024-03-25T18:02:55.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd68bdb6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:55.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd68bdb6640 1 -- 172.21.15.18:0/3084492721 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd684130210 con 0x7fd68413f710 2024-03-25T18:02:55.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd68bdb6640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:02:55.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd68bdb6640 10 monclient: _renew_subs 2024-03-25T18:02:55.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd68bdb6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:55.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd68bdb6640 1 -- 172.21.15.18:0/3084492721 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd684138e00 con 0x7fd68413f710 2024-03-25T18:02:55.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.289+0000 7fd68bdb6640 1 librados: init done 2024-03-25T18:02:55.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.290+0000 7fd68bdb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:55.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.290+0000 7fd68bdb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:55.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.290+0000 7fd68bdb6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:55.292 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.291+0000 7fd68bdb6640 1 -- 172.21.15.18:0/3084492721 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6840b3ef0 con 0x7fd68413f710 2024-03-25T18:02:55.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.568+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd674001c70 con 0x7fd68413f710 2024-03-25T18:02:55.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.568+0000 7fd67a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:02:55.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.568+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd674003a60 con 0x7fd68413f710 2024-03-25T18:02:55.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.568+0000 7fd67a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:02:55.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.568+0000 7fd67a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:02:55.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.568+0000 7fd67bfff640 4 set_mon_vals no callback set 2024-03-25T18:02:55.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.568+0000 7fd67a7fc640 10 monclient: dump: 2024-03-25T18:02:55.569 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:02:55.569 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:02:55.569 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:55.569 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:02:55.569 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:02:55.569 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:02:55.569 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:02:55.569 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:02:55.569 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:02:55.569 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:02:55.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.569+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fd674003c80 con 0x7fd68413f710 2024-03-25T18:02:55.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.570+0000 7fd689b52640 10 monclient: get_auth_request con 0x7fd66806e020 auth_method 0 2024-03-25T18:02:55.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.570+0000 7fd67a7fc640 10 client.4510.objecter ms_handle_connect 0x7fd66806e020 2024-03-25T18:02:55.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.574+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fd6740548b0 con 0x7fd68413f710 2024-03-25T18:02:55.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.574+0000 7fd67a7fc640 10 client.4510.objecter ms_dispatch 0x7fd684000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:02:55.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.574+0000 7fd67a7fc640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:02:55.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.574+0000 7fd67a7fc640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:02:55.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.574+0000 7fd67a7fc640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-25T18:02:55.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.574+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd6740503c0 con 0x7fd68413f710 2024-03-25T18:02:55.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.574+0000 7fd67a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:02:55.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.574+0000 7fd67a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:02:55.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.841+0000 7fd68bdb6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:55.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.841+0000 7fd68bdb6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:55.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.841+0000 7fd68bdb6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:02:55.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.841+0000 7fd68bdb6640 1 -- 172.21.15.18:0/3084492721 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6840627b0 con 0x7fd68413f710 2024-03-25T18:02:55.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.875+0000 7fd67a7fc640 1 -- 172.21.15.18:0/3084492721 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fd6740522f0 con 0x7fd68413f710 2024-03-25T18:02:55.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.875+0000 7fd67a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:02:55.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.875+0000 7fd67a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:02:55.876 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:02:55.876 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:02:55.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.877+0000 7fd68bdb6640 10 librados: watch_flush enter 2024-03-25T18:02:55.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.877+0000 7fd68bdb6640 10 librados: watch_flush exit 2024-03-25T18:02:55.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.877+0000 7fd68bdb6640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-25T18:02:55.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.877+0000 7fd68bdb6640 10 client.4510.objecter successfully canceled tick 2024-03-25T18:02:55.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.878+0000 7fd68bdb6640 1 -- 172.21.15.18:0/3084492721 >> v1:172.21.15.18:6800/3721244451 conn(0x7fd66806e020 legacy=0x7fd6680704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:55.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.878+0000 7fd68bdb6640 10 monclient: shutdown 2024-03-25T18:02:55.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.878+0000 7fd68bdb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:02:55.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.878+0000 7fd68bdb6640 1 -- 172.21.15.18:0/3084492721 >> v1:172.21.15.169:6789/0 conn(0x7fd68413f710 legacy=0x7fd6841db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:02:55.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.878+0000 7fd68bdb6640 1 -- 172.21.15.18:0/3084492721 shutdown_connections 2024-03-25T18:02:55.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.878+0000 7fd68bdb6640 1 -- 172.21.15.18:0/3084492721 >> 172.21.15.18:0/3084492721 conn(0x7fd6840b7ea0 msgr2=0x7fd684074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:02:55.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.878+0000 7fd68bdb6640 1 -- 172.21.15.18:0/3084492721 shutdown_connections 2024-03-25T18:02:55.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.878+0000 7fd68bdb6640 1 -- 172.21.15.18:0/3084492721 wait complete. 2024-03-25T18:02:55.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:02:55.878+0000 7fd68bdb6640 1 librados: shutdown 2024-03-25T18:02:55.891 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:03:02.697 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:03:02.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.813+0000 7fceb9b23640 10 monclient: get_monmap_and_config 2024-03-25T18:03:02.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.813+0000 7fceb9b23640 10 monclient: build_initial_monmap 2024-03-25T18:03:02.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.813+0000 7fceb9b23640 10 monclient: monmap: 2024-03-25T18:03:02.814 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:03:02.814 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:02.814 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:03:02.814066+0000 2024-03-25T18:03:02.814 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:03:02.814066+0000 2024-03-25T18:03:02.814 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:02.814 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:02.814 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:03:02.814 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:03:02.814 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:03:02.814 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:02.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.814+0000 7fceb9b23640 1 Processor -- start 2024-03-25T18:03:02.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.814+0000 7fceb9b23640 1 -- start start 2024-03-25T18:03:02.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.814+0000 7fceb9b23640 10 monclient: init 2024-03-25T18:03:02.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.815+0000 7fceb9b23640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:02.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.815+0000 7fceb9b23640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:03:02.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.815+0000 7fceb9b23640 10 monclient(hunting): picked mon.noname-b con 0x7fceb40b4f40 addr v1:172.21.15.18:6790/0 2024-03-25T18:03:02.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.815+0000 7fceb9b23640 10 monclient(hunting): picked mon.noname-c con 0x7fceb413fd90 addr v1:172.21.15.169:6789/0 2024-03-25T18:03:02.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.815+0000 7fceb9b23640 10 monclient(hunting): picked mon.noname-a con 0x7fceb40b4540 addr v1:172.21.15.18:6789/0 2024-03-25T18:03:02.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.815+0000 7fceb9b23640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb40aa0c0 con 0x7fceb40b4540 2024-03-25T18:03:02.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.815+0000 7fceb9b23640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb40ab170 con 0x7fceb413fd90 2024-03-25T18:03:02.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.815+0000 7fceb9b23640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb40a9ba0 con 0x7fceb40b4f40 2024-03-25T18:03:02.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.815+0000 7fceb9b23640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:02.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.815+0000 7fceb9b23640 10 monclient(hunting): authenticate will time out at 7311.393481s 2024-03-25T18:03:02.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.821+0000 7fceb3fff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fceb40b4540 0x7fceb414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:38862/0 (socket says 172.21.15.18:38862) 2024-03-25T18:03:02.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.821+0000 7fceb3fff640 1 -- 172.21.15.18:0/89728486 learned_addr learned my addr 172.21.15.18:0/89728486 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:03:02.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.835+0000 7fceb27fc640 1 -- 172.21.15.18:0/89728486 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982800265 0 0) 0x7fceb40aa0c0 con 0x7fceb40b4540 2024-03-25T18:03:02.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.835+0000 7fceb27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:03:02.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.835+0000 7fceb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:02.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.835+0000 7fceb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:02.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.835+0000 7fceb27fc640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:03:02.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.835+0000 7fceb27fc640 1 -- 172.21.15.18:0/89728486 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce9c003520 con 0x7fceb40b4540 2024-03-25T18:03:02.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.849+0000 7fceb27fc640 1 -- 172.21.15.18:0/89728486 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956141932 0 0) 0x7fceb40ab170 con 0x7fceb413fd90 2024-03-25T18:03:02.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.849+0000 7fceb27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:03:02.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.849+0000 7fceb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:02.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.849+0000 7fceb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:02.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.849+0000 7fceb27fc640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:03:02.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.849+0000 7fceb27fc640 1 -- 172.21.15.18:0/89728486 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceb40aa0c0 con 0x7fceb413fd90 2024-03-25T18:03:02.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.850+0000 7fceb27fc640 1 -- 172.21.15.18:0/89728486 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1817524865 0 0) 0x7fce9c003520 con 0x7fceb40b4540 2024-03-25T18:03:02.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.850+0000 7fceb27fc640 1 -- 172.21.15.18:0/89728486 >> v1:172.21.15.18:6790/0 conn(0x7fceb40b4f40 legacy=0x7fceb413f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:02.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.850+0000 7fceb27fc640 1 -- 172.21.15.18:0/89728486 >> v1:172.21.15.169:6789/0 conn(0x7fceb413fd90 legacy=0x7fceb41423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:02.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.850+0000 7fceb27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:03:02.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.850+0000 7fceb27fc640 1 monclient: found mon.noname-a 2024-03-25T18:03:02.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.850+0000 7fceb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:02.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.850+0000 7fceb27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:03:02.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.850+0000 7fceb27fc640 1 -- 172.21.15.18:0/89728486 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb40a9680 con 0x7fceb40b4540 2024-03-25T18:03:02.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.850+0000 7fceb27fc640 10 monclient: _finish_auth 0 2024-03-25T18:03:02.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.850+0000 7fceb27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:03:02.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.850+0000 7fceb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:02.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.850+0000 7fceb27fc640 1 -- 172.21.15.18:0/89728486 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fceac001a10 con 0x7fceb40b4540 2024-03-25T18:03:02.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.850+0000 7fceb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:02.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.850+0000 7fceb27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:03:02.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.850+0000 7fceb27fc640 10 monclient: dump: 2024-03-25T18:03:02.852 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:02.852 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:02.852 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:02.852 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:02.852 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:02.852 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:02.852 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:02.852 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:02.852 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:02.852 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:02.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.850+0000 7fceb9b23640 5 monclient: authenticate success, global_id 4527 2024-03-25T18:03:02.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.850+0000 7fceb9b23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:02.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.854+0000 7fceb27fc640 1 -- 172.21.15.18:0/89728486 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fceac001b70 con 0x7fceb40b4540 2024-03-25T18:03:02.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.854+0000 7fceb27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:02.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.854+0000 7fceb27fc640 1 -- 172.21.15.18:0/89728486 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fceac001f80 con 0x7fceb40b4540 2024-03-25T18:03:02.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.854+0000 7fceb9b23640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:02.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.854+0000 7fceb9b23640 4 set_mon_vals no callback set 2024-03-25T18:03:02.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.854+0000 7fceb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:02.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.854+0000 7fceb27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:02.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.854+0000 7fceb27fc640 10 monclient: dump: 2024-03-25T18:03:02.854 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:02.854 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:02.855 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:02.855 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:02.855 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:02.855 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:02.855 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:02.855 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:02.855 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:02.855 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:02.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.854+0000 7fceb9b23640 10 monclient: shutdown 2024-03-25T18:03:02.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.854+0000 7fceb9b23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:02.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.854+0000 7fceb9b23640 1 -- 172.21.15.18:0/89728486 >> v1:172.21.15.18:6789/0 conn(0x7fceb40b4540 legacy=0x7fceb414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:02.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.855+0000 7fceb9b23640 1 -- 172.21.15.18:0/89728486 shutdown_connections 2024-03-25T18:03:02.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.855+0000 7fceb9b23640 1 -- 172.21.15.18:0/89728486 >> 172.21.15.18:0/89728486 conn(0x7fceb40617a0 msgr2=0x7fceb4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:02.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.855+0000 7fceb9b23640 1 -- 172.21.15.18:0/89728486 shutdown_connections 2024-03-25T18:03:02.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.855+0000 7fceb9b23640 1 -- 172.21.15.18:0/89728486 wait complete. 2024-03-25T18:03:02.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.855+0000 7fceb9b23640 10 monclient: build_initial_monmap 2024-03-25T18:03:02.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.855+0000 7fceb9b23640 10 monclient: monmap: 2024-03-25T18:03:02.858 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:03:02.858 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:02.858 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:03:02.858 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:03:02.858 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:02.858 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:03:02.858 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:03:02.858 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:03:02.859 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:03:02.859 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:02.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.856+0000 7fceb9b23640 1 librados: starting msgr at 2024-03-25T18:03:02.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.856+0000 7fceb9b23640 1 librados: starting objecter 2024-03-25T18:03:02.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.856+0000 7fceb9b23640 1 Processor -- start 2024-03-25T18:03:02.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.856+0000 7fceb9b23640 1 -- start start 2024-03-25T18:03:02.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.856+0000 7fceb9b23640 1 librados: setting wanted keys 2024-03-25T18:03:02.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.856+0000 7fceb9b23640 1 librados: calling monclient init 2024-03-25T18:03:02.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.856+0000 7fceb9b23640 10 monclient: init 2024-03-25T18:03:02.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.857+0000 7fceb9b23640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:02.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.857+0000 7fceb9b23640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:03:02.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.857+0000 7fceb9b23640 10 monclient(hunting): picked mon.noname-a con 0x7fceb40b4540 addr v1:172.21.15.18:6789/0 2024-03-25T18:03:02.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.857+0000 7fceb9b23640 10 monclient(hunting): picked mon.noname-c con 0x7fceb40b4f40 addr v1:172.21.15.18:6790/0 2024-03-25T18:03:02.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.857+0000 7fceb9b23640 10 monclient(hunting): picked mon.noname-b con 0x7fceb413fd90 addr v1:172.21.15.169:6789/0 2024-03-25T18:03:02.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.857+0000 7fceb9b23640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb4142030 con 0x7fceb40b4540 2024-03-25T18:03:02.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.857+0000 7fceb9b23640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb4141aa0 con 0x7fceb413fd90 2024-03-25T18:03:02.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.857+0000 7fceb9b23640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb4141c30 con 0x7fceb40b4f40 2024-03-25T18:03:02.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.857+0000 7fceb9b23640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:02.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.857+0000 7fceb9b23640 10 monclient(hunting): authenticate will time out at 7311.435303s 2024-03-25T18:03:02.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.868+0000 7fceb2ffd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fceb40b4f40 0x7fceb41d7eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:55710/0 (socket says 172.21.15.18:55710) 2024-03-25T18:03:02.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.868+0000 7fceb2ffd640 1 -- 172.21.15.18:0/3760744971 learned_addr learned my addr 172.21.15.18:0/3760744971 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:03:02.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.869+0000 7fce9bfff640 10 client.?.objecter ms_handle_connect 0x7fceb40b4f40 2024-03-25T18:03:02.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.869+0000 7fce9bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:02.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.872+0000 7fce9bfff640 10 client.?.objecter ms_handle_connect 0x7fceb413fd90 2024-03-25T18:03:02.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.872+0000 7fce9bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:02.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.878+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814238729 0 0) 0x7fceb4141c30 con 0x7fceb40b4f40 2024-03-25T18:03:02.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.878+0000 7fce9bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:03:02.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.878+0000 7fce9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:02.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.878+0000 7fce9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:02.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.878+0000 7fce9bfff640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:03:02.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.878+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea0004eb0 con 0x7fceb40b4f40 2024-03-25T18:03:02.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.887+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895895425 0 0) 0x7fceb4141aa0 con 0x7fceb413fd90 2024-03-25T18:03:02.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.887+0000 7fce9bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:03:02.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.887+0000 7fce9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:02.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.887+0000 7fce9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:02.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.887+0000 7fce9bfff640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:03:02.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.887+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceb4141c30 con 0x7fceb413fd90 2024-03-25T18:03:02.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.888+0000 7fce9bfff640 10 client.?.objecter ms_handle_connect 0x7fceb40b4540 2024-03-25T18:03:02.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.888+0000 7fce9bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:02.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.896+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4060783147 0 0) 0x7fcea0004eb0 con 0x7fceb40b4f40 2024-03-25T18:03:02.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.896+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fceb4141aa0 con 0x7fceb40b4f40 2024-03-25T18:03:02.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.896+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fcea4001990 con 0x7fceb40b4f40 2024-03-25T18:03:02.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.896+0000 7fce9bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:02.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.896+0000 7fce9bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:02.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.896+0000 7fce9bfff640 10 monclient(hunting): dump: 2024-03-25T18:03:02.897 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:02.897 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:02.897 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:02.897 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:02.897 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:02.897 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:02.897 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:02.897 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:02.897 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:02.897 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:02.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.896+0000 7fce9bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:02.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.907+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2438259904 0 0) 0x7fceb4142030 con 0x7fceb40b4540 2024-03-25T18:03:02.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.907+0000 7fce9bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:03:02.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.907+0000 7fce9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:02.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.907+0000 7fce9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:02.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.907+0000 7fce9bfff640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:03:02.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.907+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea0004eb0 con 0x7fceb40b4540 2024-03-25T18:03:02.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.907+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1565875248 0 0) 0x7fceb4141c30 con 0x7fceb413fd90 2024-03-25T18:03:02.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.907+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fceb4142030 con 0x7fceb413fd90 2024-03-25T18:03:02.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.907+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fceac001e80 con 0x7fceb413fd90 2024-03-25T18:03:02.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.907+0000 7fce9bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:02.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.907+0000 7fce9bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:02.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.907+0000 7fce9bfff640 10 monclient(hunting): dump: 2024-03-25T18:03:02.908 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:02.908 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:02.908 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:02.909 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:02.909 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:02.909 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:02.909 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:02.909 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:02.909 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:02.909 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:02.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.908+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 634443827 0 0) 0x7fceb4141aa0 con 0x7fceb40b4f40 2024-03-25T18:03:02.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.908+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 >> v1:172.21.15.169:6789/0 conn(0x7fceb413fd90 legacy=0x7fceb41e20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:02.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.908+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 >> v1:172.21.15.18:6789/0 conn(0x7fceb40b4540 legacy=0x7fceb41d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:02.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.908+0000 7fce9bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:03:02.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.908+0000 7fce9bfff640 1 monclient: found mon.c 2024-03-25T18:03:02.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.908+0000 7fce9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:02.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.908+0000 7fce9bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:02.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.908+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb4141dc0 con 0x7fceb40b4f40 2024-03-25T18:03:02.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.908+0000 7fce9bfff640 10 monclient: _finish_auth 0 2024-03-25T18:03:02.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.908+0000 7fce9bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:03:02.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.908+0000 7fce9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:02.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.908+0000 7fceb9b23640 5 monclient: authenticate success, global_id 4538 2024-03-25T18:03:02.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.908+0000 7fceb9b23640 10 monclient: _renew_subs 2024-03-25T18:03:02.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.908+0000 7fceb9b23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:02.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.908+0000 7fceb9b23640 1 -- 172.21.15.18:0/3760744971 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb413d4a0 con 0x7fceb40b4f40 2024-03-25T18:03:02.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.909+0000 7fceb9b23640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:02.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.909+0000 7fceb9b23640 10 monclient: _renew_subs 2024-03-25T18:03:02.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.909+0000 7fceb9b23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:02.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.909+0000 7fceb9b23640 1 -- 172.21.15.18:0/3760744971 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb413ee00 con 0x7fceb40b4f40 2024-03-25T18:03:02.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.909+0000 7fceb9b23640 1 librados: init done 2024-03-25T18:03:02.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.910+0000 7fceb9b23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:02.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.910+0000 7fceb9b23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:02.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.910+0000 7fceb9b23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:02.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:02.910+0000 7fceb9b23640 1 -- 172.21.15.18:0/3760744971 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb4068290 con 0x7fceb40b4f40 2024-03-25T18:03:03.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.224+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcea4003610 con 0x7fceb40b4f40 2024-03-25T18:03:03.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.224+0000 7fce9bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:03.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.224+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fcea4003920 con 0x7fceb40b4f40 2024-03-25T18:03:03.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.224+0000 7fce9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:03.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.224+0000 7fce9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:03.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.224+0000 7fce9bfff640 10 monclient: dump: 2024-03-25T18:03:03.225 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:03.225 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:03.225 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:03.225 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:03.225 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:03.225 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:03.225 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:03.225 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:03.225 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:03.225 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:03.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.224+0000 7fceb1ffb640 4 set_mon_vals no callback set 2024-03-25T18:03:03.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.224+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fcea4003b40 con 0x7fceb40b4f40 2024-03-25T18:03:03.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.225+0000 7fceb37fe640 10 monclient: get_auth_request con 0x7fcea006df70 auth_method 0 2024-03-25T18:03:03.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.225+0000 7fce9bfff640 10 client.4538.objecter ms_handle_connect 0x7fcea006df70 2024-03-25T18:03:03.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.232+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fcea4054710 con 0x7fceb40b4f40 2024-03-25T18:03:03.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.233+0000 7fce9bfff640 10 client.4538.objecter ms_dispatch 0x7fceb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:03:03.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.233+0000 7fce9bfff640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:03:03.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.233+0000 7fce9bfff640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:03:03.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.233+0000 7fce9bfff640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-25T18:03:03.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.233+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcea4050220 con 0x7fceb40b4f40 2024-03-25T18:03:03.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.233+0000 7fce9bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:03.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.233+0000 7fce9bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:03.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.448+0000 7fceb9b23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:03.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.448+0000 7fceb9b23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:03.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.448+0000 7fceb9b23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:03.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.448+0000 7fceb9b23640 1 -- 172.21.15.18:0/3760744971 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fceb413f100 con 0x7fceb40b4f40 2024-03-25T18:03:03.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.489+0000 7fce9bfff640 1 -- 172.21.15.18:0/3760744971 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fceb413f100 con 0x7fceb40b4f40 2024-03-25T18:03:03.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.489+0000 7fce9bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:03.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.489+0000 7fce9bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:03.490 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:03:03.490 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:03:03.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.492+0000 7fceb9b23640 10 librados: watch_flush enter 2024-03-25T18:03:03.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.492+0000 7fceb9b23640 10 librados: watch_flush exit 2024-03-25T18:03:03.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.492+0000 7fceb9b23640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-25T18:03:03.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.492+0000 7fceb9b23640 10 client.4538.objecter successfully canceled tick 2024-03-25T18:03:03.493 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.492+0000 7fceb9b23640 1 -- 172.21.15.18:0/3760744971 >> v1:172.21.15.18:6800/3721244451 conn(0x7fcea006df70 legacy=0x7fcea0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:03.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.493+0000 7fceb9b23640 10 monclient: shutdown 2024-03-25T18:03:03.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.493+0000 7fceb9b23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:03.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.493+0000 7fceb9b23640 1 -- 172.21.15.18:0/3760744971 >> v1:172.21.15.18:6790/0 conn(0x7fceb40b4f40 legacy=0x7fceb41d7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:03.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.493+0000 7fceb9b23640 1 -- 172.21.15.18:0/3760744971 shutdown_connections 2024-03-25T18:03:03.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.493+0000 7fceb9b23640 1 -- 172.21.15.18:0/3760744971 >> 172.21.15.18:0/3760744971 conn(0x7fceb40617a0 msgr2=0x7fceb4139d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:03.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.493+0000 7fceb9b23640 1 -- 172.21.15.18:0/3760744971 shutdown_connections 2024-03-25T18:03:03.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.493+0000 7fceb9b23640 1 -- 172.21.15.18:0/3760744971 wait complete. 2024-03-25T18:03:03.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:03.493+0000 7fceb9b23640 1 librados: shutdown 2024-03-25T18:03:03.505 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:03:08.295 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:06.087 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:08.303 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:06.096 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:08.338 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:06.131 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:08.364 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:06.113 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:08.365 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:06.158 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:08.368 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:06.117 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:08.373 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:06.166 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:08.402 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:06.151 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:08.407 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:06.200 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:10.312 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:03:10.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.431+0000 7ff8381cf640 10 monclient: get_monmap_and_config 2024-03-25T18:03:10.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.431+0000 7ff8381cf640 10 monclient: build_initial_monmap 2024-03-25T18:03:10.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.431+0000 7ff8381cf640 10 monclient: monmap: 2024-03-25T18:03:10.433 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:03:10.433 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:10.433 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:03:10.432829+0000 2024-03-25T18:03:10.433 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:03:10.432829+0000 2024-03-25T18:03:10.433 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:10.433 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:10.433 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:03:10.433 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:03:10.433 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:03:10.433 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:10.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.432+0000 7ff8381cf640 1 Processor -- start 2024-03-25T18:03:10.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.432+0000 7ff8381cf640 1 -- start start 2024-03-25T18:03:10.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.432+0000 7ff8381cf640 10 monclient: init 2024-03-25T18:03:10.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.432+0000 7ff8381cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:10.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.432+0000 7ff8381cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:03:10.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.433+0000 7ff8381cf640 10 monclient(hunting): picked mon.noname-a con 0x7ff83006ab80 addr v1:172.21.15.18:6789/0 2024-03-25T18:03:10.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.433+0000 7ff8381cf640 10 monclient(hunting): picked mon.noname-b con 0x7ff83006af60 addr v1:172.21.15.18:6790/0 2024-03-25T18:03:10.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.433+0000 7ff8381cf640 10 monclient(hunting): picked mon.noname-c con 0x7ff8300b00b0 addr v1:172.21.15.169:6789/0 2024-03-25T18:03:10.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.433+0000 7ff8381cf640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8300b0500 con 0x7ff83006ab80 2024-03-25T18:03:10.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.433+0000 7ff8381cf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83013ffa0 con 0x7ff8300b00b0 2024-03-25T18:03:10.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.433+0000 7ff8381cf640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8300af520 con 0x7ff83006af60 2024-03-25T18:03:10.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.433+0000 7ff8381cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:10.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.433+0000 7ff8381cf640 10 monclient(hunting): authenticate will time out at 7319.011746s 2024-03-25T18:03:10.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.442+0000 7ff83676c640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7ff8300b00b0 0x7ff83014dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:58162/0 (socket says 172.21.15.18:58162) 2024-03-25T18:03:10.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.442+0000 7ff83676c640 1 -- 172.21.15.18:0/905571412 learned_addr learned my addr 172.21.15.18:0/905571412 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:03:10.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.459+0000 7ff834f69640 1 -- 172.21.15.18:0/905571412 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711886045 0 0) 0x7ff8300af520 con 0x7ff83006af60 2024-03-25T18:03:10.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.459+0000 7ff834f69640 10 monclient(hunting): _negotiate 2024-03-25T18:03:10.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.459+0000 7ff834f69640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:10.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.459+0000 7ff834f69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:10.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.459+0000 7ff834f69640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:03:10.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.459+0000 7ff834f69640 1 -- 172.21.15.18:0/905571412 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff81c003520 con 0x7ff83006af60 2024-03-25T18:03:10.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.460+0000 7ff834f69640 1 -- 172.21.15.18:0/905571412 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060565085 0 0) 0x7ff83013ffa0 con 0x7ff8300b00b0 2024-03-25T18:03:10.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.460+0000 7ff834f69640 10 monclient(hunting): _negotiate 2024-03-25T18:03:10.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.460+0000 7ff834f69640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:10.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.460+0000 7ff834f69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:10.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.460+0000 7ff834f69640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:03:10.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.460+0000 7ff834f69640 1 -- 172.21.15.18:0/905571412 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8300af520 con 0x7ff8300b00b0 2024-03-25T18:03:10.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.473+0000 7ff834f69640 1 -- 172.21.15.18:0/905571412 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092959260 0 0) 0x7ff8300b0500 con 0x7ff83006ab80 2024-03-25T18:03:10.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.473+0000 7ff834f69640 10 monclient(hunting): _negotiate 2024-03-25T18:03:10.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.473+0000 7ff834f69640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:10.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.473+0000 7ff834f69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:10.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.473+0000 7ff834f69640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:03:10.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.473+0000 7ff834f69640 1 -- 172.21.15.18:0/905571412 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff83013ffa0 con 0x7ff83006ab80 2024-03-25T18:03:10.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff834f69640 1 -- 172.21.15.18:0/905571412 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4169057161 0 0) 0x7ff81c003520 con 0x7ff83006af60 2024-03-25T18:03:10.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff834f69640 1 -- 172.21.15.18:0/905571412 >> v1:172.21.15.169:6789/0 conn(0x7ff8300b00b0 legacy=0x7ff83014dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:10.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff834f69640 1 -- 172.21.15.18:0/905571412 >> v1:172.21.15.18:6789/0 conn(0x7ff83006ab80 legacy=0x7ff8300ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:10.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff834f69640 10 monclient: _finish_hunting 0 2024-03-25T18:03:10.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff834f69640 1 monclient: found mon.noname-b 2024-03-25T18:03:10.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff834f69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:10.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff834f69640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:03:10.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff834f69640 1 -- 172.21.15.18:0/905571412 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff830072890 con 0x7ff83006af60 2024-03-25T18:03:10.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff834f69640 10 monclient: _finish_auth 0 2024-03-25T18:03:10.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff834f69640 10 monclient: _check_auth_tickets 2024-03-25T18:03:10.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff834f69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:10.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff834f69640 1 -- 172.21.15.18:0/905571412 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ff820001690 con 0x7ff83006af60 2024-03-25T18:03:10.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff8381cf640 5 monclient: authenticate success, global_id 4541 2024-03-25T18:03:10.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff8381cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:10.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff834f69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:10.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff834f69640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:03:10.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff834f69640 10 monclient: dump: 2024-03-25T18:03:10.481 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:10.481 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:10.481 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:10.481 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:10.482 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:10.482 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:10.482 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:10.482 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:10.482 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:10.482 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:10.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.479+0000 7ff8381cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:10.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.500+0000 7ff834f69640 1 -- 172.21.15.18:0/905571412 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8200017f0 con 0x7ff83006af60 2024-03-25T18:03:10.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.500+0000 7ff834f69640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:10.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.500+0000 7ff834f69640 1 -- 172.21.15.18:0/905571412 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ff820001c00 con 0x7ff83006af60 2024-03-25T18:03:10.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.500+0000 7ff8381cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:10.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.500+0000 7ff8381cf640 4 set_mon_vals no callback set 2024-03-25T18:03:10.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.500+0000 7ff834f69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:10.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.500+0000 7ff834f69640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:10.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.500+0000 7ff834f69640 10 monclient: dump: 2024-03-25T18:03:10.502 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:10.502 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:10.502 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:10.502 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:10.502 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:10.502 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:10.503 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:10.503 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:10.503 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:10.503 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:10.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.501+0000 7ff8381cf640 10 monclient: shutdown 2024-03-25T18:03:10.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.501+0000 7ff8381cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:10.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.501+0000 7ff8381cf640 1 -- 172.21.15.18:0/905571412 >> v1:172.21.15.18:6790/0 conn(0x7ff83006af60 legacy=0x7ff83013f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:10.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.501+0000 7ff8381cf640 1 -- 172.21.15.18:0/905571412 shutdown_connections 2024-03-25T18:03:10.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.501+0000 7ff8381cf640 1 -- 172.21.15.18:0/905571412 >> 172.21.15.18:0/905571412 conn(0x7ff8300b4a80 msgr2=0x7ff8300b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:10.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.501+0000 7ff8381cf640 1 -- 172.21.15.18:0/905571412 shutdown_connections 2024-03-25T18:03:10.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.501+0000 7ff8381cf640 1 -- 172.21.15.18:0/905571412 wait complete. 2024-03-25T18:03:10.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.502+0000 7ff8381cf640 10 monclient: build_initial_monmap 2024-03-25T18:03:10.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.502+0000 7ff8381cf640 10 monclient: monmap: 2024-03-25T18:03:10.505 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:03:10.505 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:10.506 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:03:10.506 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:03:10.506 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:10.506 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:03:10.506 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:03:10.506 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:03:10.506 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:03:10.506 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:10.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.502+0000 7ff8381cf640 1 librados: starting msgr at 2024-03-25T18:03:10.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.502+0000 7ff8381cf640 1 librados: starting objecter 2024-03-25T18:03:10.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.502+0000 7ff8381cf640 1 Processor -- start 2024-03-25T18:03:10.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.502+0000 7ff8381cf640 1 -- start start 2024-03-25T18:03:10.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.502+0000 7ff8381cf640 1 librados: setting wanted keys 2024-03-25T18:03:10.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.502+0000 7ff8381cf640 1 librados: calling monclient init 2024-03-25T18:03:10.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.502+0000 7ff8381cf640 10 monclient: init 2024-03-25T18:03:10.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.503+0000 7ff8381cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:10.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.503+0000 7ff8381cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:03:10.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.503+0000 7ff8381cf640 10 monclient(hunting): picked mon.noname-c con 0x7ff83006ab80 addr v1:172.21.15.18:6790/0 2024-03-25T18:03:10.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.503+0000 7ff8381cf640 10 monclient(hunting): picked mon.noname-a con 0x7ff8300b00b0 addr v1:172.21.15.18:6789/0 2024-03-25T18:03:10.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.503+0000 7ff8381cf640 10 monclient(hunting): picked mon.noname-b con 0x7ff8301db8f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:03:10.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.503+0000 7ff8381cf640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff830145860 con 0x7ff8300b00b0 2024-03-25T18:03:10.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.503+0000 7ff8381cf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8301db3d0 con 0x7ff8301db8f0 2024-03-25T18:03:10.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.503+0000 7ff8381cf640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8301431b0 con 0x7ff83006ab80 2024-03-25T18:03:10.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.503+0000 7ff8381cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:10.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.503+0000 7ff8381cf640 10 monclient(hunting): authenticate will time out at 7319.082210s 2024-03-25T18:03:10.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.512+0000 7ff835f6b640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7ff83006ab80 0x7ff830145fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:40328/0 (socket says 172.21.15.18:40328) 2024-03-25T18:03:10.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.512+0000 7ff835f6b640 1 -- 172.21.15.18:0/1126791589 learned_addr learned my addr 172.21.15.18:0/1126791589 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:03:10.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.514+0000 7ff8267fc640 10 client.?.objecter ms_handle_connect 0x7ff83006ab80 2024-03-25T18:03:10.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.514+0000 7ff8267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:10.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.514+0000 7ff8267fc640 10 client.?.objecter ms_handle_connect 0x7ff8301db8f0 2024-03-25T18:03:10.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.514+0000 7ff8267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:10.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.515+0000 7ff8267fc640 10 client.?.objecter ms_handle_connect 0x7ff8300b00b0 2024-03-25T18:03:10.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.515+0000 7ff8267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:10.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.525+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740982672 0 0) 0x7ff8301431b0 con 0x7ff83006ab80 2024-03-25T18:03:10.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.525+0000 7ff8267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:03:10.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.525+0000 7ff8267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:10.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.525+0000 7ff8267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:10.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.525+0000 7ff8267fc640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:03:10.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.525+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff814004eb0 con 0x7ff83006ab80 2024-03-25T18:03:10.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.529+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779369565 0 0) 0x7ff8301db3d0 con 0x7ff8301db8f0 2024-03-25T18:03:10.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.529+0000 7ff8267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:03:10.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.529+0000 7ff8267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:10.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.529+0000 7ff8267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:10.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.529+0000 7ff8267fc640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:03:10.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.529+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8301431b0 con 0x7ff8301db8f0 2024-03-25T18:03:10.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.537+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667533557 0 0) 0x7ff830145860 con 0x7ff8300b00b0 2024-03-25T18:03:10.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.537+0000 7ff8267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:03:10.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.537+0000 7ff8267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:10.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.537+0000 7ff8267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:10.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.537+0000 7ff8267fc640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:03:10.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.537+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8301db3d0 con 0x7ff8300b00b0 2024-03-25T18:03:10.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.545+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3872853 0 0) 0x7ff8301db3d0 con 0x7ff8300b00b0 2024-03-25T18:03:10.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.545+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff830145860 con 0x7ff8300b00b0 2024-03-25T18:03:10.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.545+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ff820001b00 con 0x7ff8300b00b0 2024-03-25T18:03:10.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.545+0000 7ff8267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:10.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.545+0000 7ff8267fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:03:10.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.545+0000 7ff8267fc640 10 monclient(hunting): dump: 2024-03-25T18:03:10.547 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:10.547 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:10.547 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:10.547 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:10.547 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:10.547 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:10.547 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:10.547 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:10.547 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:10.547 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:10.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.545+0000 7ff8267fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:10.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.546+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3952376786 0 0) 0x7ff814004eb0 con 0x7ff83006ab80 2024-03-25T18:03:10.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.546+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8301db3d0 con 0x7ff83006ab80 2024-03-25T18:03:10.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.546+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ff82c001a10 con 0x7ff83006ab80 2024-03-25T18:03:10.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.546+0000 7ff8267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:10.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.546+0000 7ff8267fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:10.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.546+0000 7ff8267fc640 10 monclient(hunting): dump: 2024-03-25T18:03:10.549 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:10.549 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:10.549 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:10.549 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:10.549 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:10.549 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:10.549 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:10.549 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:10.549 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:10.549 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:10.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.547+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1368775824 0 0) 0x7ff8301431b0 con 0x7ff8301db8f0 2024-03-25T18:03:10.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.547+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff814004eb0 con 0x7ff8301db8f0 2024-03-25T18:03:10.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.548+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ff828001b50 con 0x7ff8301db8f0 2024-03-25T18:03:10.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.548+0000 7ff8267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:10.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.548+0000 7ff8267fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:10.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.548+0000 7ff8267fc640 10 monclient(hunting): dump: 2024-03-25T18:03:10.550 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:10.550 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:10.550 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:10.550 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:10.550 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:10.551 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:10.551 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:10.551 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:10.551 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:10.551 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:10.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.559+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1434962759 0 0) 0x7ff814004eb0 con 0x7ff8301db8f0 2024-03-25T18:03:10.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.559+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 >> v1:172.21.15.18:6790/0 conn(0x7ff83006ab80 legacy=0x7ff830145fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:10.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.559+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 >> v1:172.21.15.18:6789/0 conn(0x7ff8300b00b0 legacy=0x7ff830143460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:10.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.559+0000 7ff8267fc640 10 monclient: _finish_hunting 0 2024-03-25T18:03:10.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.559+0000 7ff8267fc640 1 monclient: found mon.b 2024-03-25T18:03:10.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.559+0000 7ff8267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:10.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.559+0000 7ff8267fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:03:10.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.559+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8301466e0 con 0x7ff8301db8f0 2024-03-25T18:03:10.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.560+0000 7ff8267fc640 10 monclient: _finish_auth 0 2024-03-25T18:03:10.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.560+0000 7ff8267fc640 10 monclient: _check_auth_tickets 2024-03-25T18:03:10.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.560+0000 7ff8267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:10.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.560+0000 7ff8381cf640 5 monclient: authenticate success, global_id 4522 2024-03-25T18:03:10.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.560+0000 7ff8381cf640 10 monclient: _renew_subs 2024-03-25T18:03:10.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.560+0000 7ff8381cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:03:10.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.560+0000 7ff8381cf640 1 -- 172.21.15.18:0/1126791589 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff830143b70 con 0x7ff8301db8f0 2024-03-25T18:03:10.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.560+0000 7ff8381cf640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:10.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.560+0000 7ff8381cf640 10 monclient: _renew_subs 2024-03-25T18:03:10.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.560+0000 7ff8381cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:03:10.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.560+0000 7ff8381cf640 1 -- 172.21.15.18:0/1126791589 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8301d5100 con 0x7ff8301db8f0 2024-03-25T18:03:10.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.560+0000 7ff8381cf640 1 librados: init done 2024-03-25T18:03:10.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.561+0000 7ff8381cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:10.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.561+0000 7ff8381cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:10.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.561+0000 7ff8381cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:03:10.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.561+0000 7ff8381cf640 1 -- 172.21.15.18:0/1126791589 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8301db560 con 0x7ff8301db8f0 2024-03-25T18:03:10.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.849+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff828002080 con 0x7ff8301db8f0 2024-03-25T18:03:10.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.849+0000 7ff8267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:10.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.849+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ff828004d00 con 0x7ff8301db8f0 2024-03-25T18:03:10.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.849+0000 7ff8267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:10.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.849+0000 7ff8267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:10.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.849+0000 7ff8267fc640 10 monclient: dump: 2024-03-25T18:03:10.851 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:10.851 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:10.851 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:10.851 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:10.851 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:10.851 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:10.851 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:10.851 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:10.851 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:10.852 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:10.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.849+0000 7ff827fff640 4 set_mon_vals no callback set 2024-03-25T18:03:10.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.850+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7ff82801ad10 con 0x7ff8301db8f0 2024-03-25T18:03:10.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.850+0000 7ff835f6b640 10 monclient: get_auth_request con 0x7ff81406e0e0 auth_method 0 2024-03-25T18:03:10.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.851+0000 7ff8267fc640 10 client.4522.objecter ms_handle_connect 0x7ff81406e0e0 2024-03-25T18:03:10.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.855+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7ff828054a20 con 0x7ff8301db8f0 2024-03-25T18:03:10.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.855+0000 7ff8267fc640 10 client.4522.objecter ms_dispatch 0x7ff830000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:03:10.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.855+0000 7ff8267fc640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:03:10.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.855+0000 7ff8267fc640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:03:10.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.855+0000 7ff8267fc640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-25T18:03:10.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.855+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff828050360 con 0x7ff8301db8f0 2024-03-25T18:03:10.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.855+0000 7ff8267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:10.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:10.855+0000 7ff8267fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:11.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.110+0000 7ff8381cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:11.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.110+0000 7ff8381cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:11.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.110+0000 7ff8381cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:03:11.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.110+0000 7ff8381cf640 1 -- 172.21.15.18:0/1126791589 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8301db1c0 con 0x7ff8301db8f0 2024-03-25T18:03:11.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.191+0000 7ff8267fc640 1 -- 172.21.15.18:0/1126791589 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7ff828052460 con 0x7ff8301db8f0 2024-03-25T18:03:11.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.191+0000 7ff8267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:11.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.191+0000 7ff8267fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:11.193 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:03:11.193 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:03:11.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.193+0000 7ff8381cf640 10 librados: watch_flush enter 2024-03-25T18:03:11.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.194+0000 7ff8381cf640 10 librados: watch_flush exit 2024-03-25T18:03:11.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.194+0000 7ff8381cf640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-25T18:03:11.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.194+0000 7ff8381cf640 10 client.4522.objecter successfully canceled tick 2024-03-25T18:03:11.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.194+0000 7ff8381cf640 1 -- 172.21.15.18:0/1126791589 >> v1:172.21.15.18:6800/3721244451 conn(0x7ff81406e0e0 legacy=0x7ff8140705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:11.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.194+0000 7ff8381cf640 10 monclient: shutdown 2024-03-25T18:03:11.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.194+0000 7ff8381cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:11.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.194+0000 7ff8381cf640 1 -- 172.21.15.18:0/1126791589 >> v1:172.21.15.169:6789/0 conn(0x7ff8301db8f0 legacy=0x7ff8301d4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:11.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.194+0000 7ff8381cf640 1 -- 172.21.15.18:0/1126791589 shutdown_connections 2024-03-25T18:03:11.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.194+0000 7ff8381cf640 1 -- 172.21.15.18:0/1126791589 >> 172.21.15.18:0/1126791589 conn(0x7ff8300b4a80 msgr2=0x7ff83006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:11.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.194+0000 7ff8381cf640 1 -- 172.21.15.18:0/1126791589 shutdown_connections 2024-03-25T18:03:11.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.194+0000 7ff8381cf640 1 -- 172.21.15.18:0/1126791589 wait complete. 2024-03-25T18:03:11.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:11.194+0000 7ff8381cf640 1 librados: shutdown 2024-03-25T18:03:11.206 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:03:18.013 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:03:18.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.129+0000 7fba76414640 10 monclient: get_monmap_and_config 2024-03-25T18:03:18.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.129+0000 7fba76414640 10 monclient: build_initial_monmap 2024-03-25T18:03:18.131 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.129+0000 7fba76414640 10 monclient: monmap: 2024-03-25T18:03:18.131 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:03:18.131 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:18.131 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:03:18.131329+0000 2024-03-25T18:03:18.131 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:03:18.131329+0000 2024-03-25T18:03:18.132 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:18.132 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:18.132 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:03:18.132 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:03:18.132 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:03:18.132 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:18.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.130+0000 7fba76414640 1 Processor -- start 2024-03-25T18:03:18.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.130+0000 7fba76414640 1 -- start start 2024-03-25T18:03:18.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.130+0000 7fba76414640 10 monclient: init 2024-03-25T18:03:18.133 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.131+0000 7fba76414640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:18.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.131+0000 7fba76414640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:03:18.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.131+0000 7fba76414640 10 monclient(hunting): picked mon.noname-a con 0x7fba700b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T18:03:18.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.131+0000 7fba76414640 10 monclient(hunting): picked mon.noname-c con 0x7fba700b1730 addr v1:172.21.15.169:6789/0 2024-03-25T18:03:18.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.131+0000 7fba76414640 10 monclient(hunting): picked mon.noname-b con 0x7fba7014be90 addr v1:172.21.15.18:6790/0 2024-03-25T18:03:18.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.131+0000 7fba76414640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba700af940 con 0x7fba700b4d60 2024-03-25T18:03:18.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.131+0000 7fba76414640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba700b0a70 con 0x7fba700b1730 2024-03-25T18:03:18.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.131+0000 7fba76414640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba700b0d50 con 0x7fba7014be90 2024-03-25T18:03:18.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.131+0000 7fba76414640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:18.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.131+0000 7fba76414640 10 monclient(hunting): authenticate will time out at 7326.710522s 2024-03-25T18:03:18.139 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.138+0000 7fba6f7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fba700b1730 0x7fba700b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:54526/0 (socket says 172.21.15.18:54526) 2024-03-25T18:03:18.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.138+0000 7fba6f7fe640 1 -- 172.21.15.18:0/2330429365 learned_addr learned my addr 172.21.15.18:0/2330429365 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:03:18.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.154+0000 7fba6effd640 1 -- 172.21.15.18:0/2330429365 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677002692 0 0) 0x7fba700b0a70 con 0x7fba700b1730 2024-03-25T18:03:18.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.154+0000 7fba6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:18.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.154+0000 7fba6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:18.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.154+0000 7fba6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:18.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.154+0000 7fba6effd640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:03:18.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.154+0000 7fba6effd640 1 -- 172.21.15.18:0/2330429365 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba58003520 con 0x7fba700b1730 2024-03-25T18:03:18.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.155+0000 7fba6effd640 1 -- 172.21.15.18:0/2330429365 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172460968 0 0) 0x7fba700af940 con 0x7fba700b4d60 2024-03-25T18:03:18.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.155+0000 7fba6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:18.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.155+0000 7fba6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:18.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.155+0000 7fba6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:18.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.155+0000 7fba6effd640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:03:18.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.156+0000 7fba6effd640 1 -- 172.21.15.18:0/2330429365 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba700b0a70 con 0x7fba700b4d60 2024-03-25T18:03:18.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba6effd640 1 -- 172.21.15.18:0/2330429365 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1741172019 0 0) 0x7fba700b0a70 con 0x7fba700b4d60 2024-03-25T18:03:18.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba6effd640 1 -- 172.21.15.18:0/2330429365 >> v1:172.21.15.18:6790/0 conn(0x7fba7014be90 legacy=0x7fba7014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:18.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba6effd640 1 -- 172.21.15.18:0/2330429365 >> v1:172.21.15.169:6789/0 conn(0x7fba700b1730 legacy=0x7fba700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:18.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba6effd640 10 monclient: _finish_hunting 0 2024-03-25T18:03:18.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba6effd640 1 monclient: found mon.noname-a 2024-03-25T18:03:18.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:18.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba6effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:03:18.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba6effd640 1 -- 172.21.15.18:0/2330429365 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba700b1030 con 0x7fba700b4d60 2024-03-25T18:03:18.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba6effd640 10 monclient: _finish_auth 0 2024-03-25T18:03:18.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba6effd640 10 monclient: _check_auth_tickets 2024-03-25T18:03:18.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:18.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba6effd640 1 -- 172.21.15.18:0/2330429365 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fba60001670 con 0x7fba700b4d60 2024-03-25T18:03:18.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba76414640 5 monclient: authenticate success, global_id 4539 2024-03-25T18:03:18.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba76414640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:18.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:18.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba6effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:03:18.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba6effd640 10 monclient: dump: 2024-03-25T18:03:18.165 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:18.165 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:18.165 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:18.165 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:18.165 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:18.166 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:18.166 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:18.166 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:18.166 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:18.166 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:18.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.162+0000 7fba76414640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:18.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.170+0000 7fba6effd640 1 -- 172.21.15.18:0/2330429365 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba600017d0 con 0x7fba700b4d60 2024-03-25T18:03:18.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.170+0000 7fba6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:18.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.170+0000 7fba6effd640 1 -- 172.21.15.18:0/2330429365 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fba60001ba0 con 0x7fba700b4d60 2024-03-25T18:03:18.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.170+0000 7fba76414640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:18.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.170+0000 7fba76414640 4 set_mon_vals no callback set 2024-03-25T18:03:18.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.170+0000 7fba6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:18.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.170+0000 7fba6effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:18.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.170+0000 7fba6effd640 10 monclient: dump: 2024-03-25T18:03:18.172 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:18.172 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:18.172 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:18.172 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:18.172 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:18.172 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:18.173 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:18.173 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:18.173 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:18.173 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:18.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.171+0000 7fba76414640 10 monclient: shutdown 2024-03-25T18:03:18.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.171+0000 7fba76414640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:18.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.171+0000 7fba76414640 1 -- 172.21.15.18:0/2330429365 >> v1:172.21.15.18:6789/0 conn(0x7fba700b4d60 legacy=0x7fba7013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:18.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.171+0000 7fba76414640 1 -- 172.21.15.18:0/2330429365 shutdown_connections 2024-03-25T18:03:18.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.171+0000 7fba76414640 1 -- 172.21.15.18:0/2330429365 >> 172.21.15.18:0/2330429365 conn(0x7fba700617a0 msgr2=0x7fba70061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:18.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.171+0000 7fba76414640 1 -- 172.21.15.18:0/2330429365 shutdown_connections 2024-03-25T18:03:18.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.171+0000 7fba76414640 1 -- 172.21.15.18:0/2330429365 wait complete. 2024-03-25T18:03:18.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.172+0000 7fba76414640 10 monclient: build_initial_monmap 2024-03-25T18:03:18.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.172+0000 7fba76414640 10 monclient: monmap: 2024-03-25T18:03:18.175 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:03:18.175 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:18.175 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:03:18.175 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:03:18.175 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:18.175 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:03:18.175 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:03:18.175 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:03:18.175 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:03:18.175 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:18.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.172+0000 7fba76414640 1 librados: starting msgr at 2024-03-25T18:03:18.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.172+0000 7fba76414640 1 librados: starting objecter 2024-03-25T18:03:18.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.173+0000 7fba76414640 1 Processor -- start 2024-03-25T18:03:18.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.173+0000 7fba76414640 1 -- start start 2024-03-25T18:03:18.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.173+0000 7fba76414640 1 librados: setting wanted keys 2024-03-25T18:03:18.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.173+0000 7fba76414640 1 librados: calling monclient init 2024-03-25T18:03:18.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.173+0000 7fba76414640 10 monclient: init 2024-03-25T18:03:18.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.174+0000 7fba76414640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:18.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.174+0000 7fba76414640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:03:18.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.174+0000 7fba76414640 10 monclient(hunting): picked mon.noname-b con 0x7fba700b1730 addr v1:172.21.15.169:6789/0 2024-03-25T18:03:18.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.174+0000 7fba76414640 10 monclient(hunting): picked mon.noname-c con 0x7fba700b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T18:03:18.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.174+0000 7fba76414640 10 monclient(hunting): picked mon.noname-a con 0x7fba7014be90 addr v1:172.21.15.18:6789/0 2024-03-25T18:03:18.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.174+0000 7fba76414640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba701327e0 con 0x7fba7014be90 2024-03-25T18:03:18.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.174+0000 7fba76414640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba70140960 con 0x7fba700b1730 2024-03-25T18:03:18.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.174+0000 7fba76414640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba701454a0 con 0x7fba700b4d60 2024-03-25T18:03:18.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.174+0000 7fba76414640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:18.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.174+0000 7fba76414640 10 monclient(hunting): authenticate will time out at 7326.752909s 2024-03-25T18:03:18.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.180+0000 7fba749b1640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fba7014be90 0x7fba701320d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:57366/0 (socket says 172.21.15.18:57366) 2024-03-25T18:03:18.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.180+0000 7fba749b1640 1 -- 172.21.15.18:0/3612885797 learned_addr learned my addr 172.21.15.18:0/3612885797 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:03:18.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.181+0000 7fba6cff9640 10 client.?.objecter ms_handle_connect 0x7fba7014be90 2024-03-25T18:03:18.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.181+0000 7fba6cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:18.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.187+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015859878 0 0) 0x7fba701327e0 con 0x7fba7014be90 2024-03-25T18:03:18.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.187+0000 7fba6cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:03:18.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.187+0000 7fba6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:18.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.187+0000 7fba6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:18.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.187+0000 7fba6cff9640 10 monclient(hunting): my global_id is 4542 2024-03-25T18:03:18.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.187+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba5c004eb0 con 0x7fba7014be90 2024-03-25T18:03:18.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.191+0000 7fba6cff9640 10 client.?.objecter ms_handle_connect 0x7fba700b1730 2024-03-25T18:03:18.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.191+0000 7fba6cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:18.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.201+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798398834 0 0) 0x7fba70140960 con 0x7fba700b1730 2024-03-25T18:03:18.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.201+0000 7fba6cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:03:18.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.201+0000 7fba6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:18.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.201+0000 7fba6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:18.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.201+0000 7fba6cff9640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:03:18.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.201+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba701327e0 con 0x7fba700b1730 2024-03-25T18:03:18.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.203+0000 7fba6cff9640 10 client.?.objecter ms_handle_connect 0x7fba700b4d60 2024-03-25T18:03:18.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.203+0000 7fba6cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:18.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.216+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2093338548 0 0) 0x7fba5c004eb0 con 0x7fba7014be90 2024-03-25T18:03:18.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.216+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba70140960 con 0x7fba7014be90 2024-03-25T18:03:18.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.216+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fba68001c30 con 0x7fba7014be90 2024-03-25T18:03:18.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.216+0000 7fba6cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:18.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.216+0000 7fba6cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:03:18.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.216+0000 7fba6cff9640 10 monclient(hunting): dump: 2024-03-25T18:03:18.218 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:18.218 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:18.218 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:18.218 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:18.218 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:18.218 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:18.218 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:18.218 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:18.218 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:18.218 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:18.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.216+0000 7fba6cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:18.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.216+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1890931612 0 0) 0x7fba701327e0 con 0x7fba700b1730 2024-03-25T18:03:18.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.216+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba5c004eb0 con 0x7fba700b1730 2024-03-25T18:03:18.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.216+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fba60001ae0 con 0x7fba700b1730 2024-03-25T18:03:18.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.216+0000 7fba6cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:18.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.216+0000 7fba6cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:18.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.216+0000 7fba6cff9640 10 monclient(hunting): dump: 2024-03-25T18:03:18.219 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:18.219 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:18.220 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:18.220 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:18.220 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:18.220 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:18.220 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:18.220 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:18.220 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:18.220 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:18.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3776986157 0 0) 0x7fba70140960 con 0x7fba7014be90 2024-03-25T18:03:18.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 >> v1:172.21.15.18:6790/0 conn(0x7fba700b4d60 legacy=0x7fba701319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:18.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 >> v1:172.21.15.169:6789/0 conn(0x7fba700b1730 legacy=0x7fba70142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:18.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba6cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:03:18.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba6cff9640 1 monclient: found mon.a 2024-03-25T18:03:18.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba6cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:18.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba6cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:03:18.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba70140db0 con 0x7fba7014be90 2024-03-25T18:03:18.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba6cff9640 10 monclient: _finish_auth 0 2024-03-25T18:03:18.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba6cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:03:18.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba6cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:18.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba76414640 5 monclient: authenticate success, global_id 4542 2024-03-25T18:03:18.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba76414640 10 monclient: _renew_subs 2024-03-25T18:03:18.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba76414640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:03:18.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba76414640 1 -- 172.21.15.18:0/3612885797 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba70140fa0 con 0x7fba7014be90 2024-03-25T18:03:18.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba76414640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:18.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba76414640 10 monclient: _renew_subs 2024-03-25T18:03:18.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba76414640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:03:18.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba76414640 1 -- 172.21.15.18:0/3612885797 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba70140560 con 0x7fba7014be90 2024-03-25T18:03:18.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.224+0000 7fba76414640 1 librados: init done 2024-03-25T18:03:18.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.225+0000 7fba76414640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:18.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.225+0000 7fba76414640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:18.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.225+0000 7fba76414640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:03:18.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.225+0000 7fba76414640 1 -- 172.21.15.18:0/3612885797 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba700627b0 con 0x7fba7014be90 2024-03-25T18:03:18.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.535+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba68003990 con 0x7fba7014be90 2024-03-25T18:03:18.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.535+0000 7fba6cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:18.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.535+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fba68003da0 con 0x7fba7014be90 2024-03-25T18:03:18.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.535+0000 7fba6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:18.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.535+0000 7fba6cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:18.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.535+0000 7fba6cff9640 10 monclient: dump: 2024-03-25T18:03:18.537 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:18.537 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:18.538 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:18.538 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:18.538 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:18.538 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:18.538 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:18.538 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:18.538 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:18.538 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:18.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.535+0000 7fba6e7fc640 4 set_mon_vals no callback set 2024-03-25T18:03:18.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.536+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fba68003fc0 con 0x7fba7014be90 2024-03-25T18:03:18.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.537+0000 7fba6ffff640 10 monclient: get_auth_request con 0x7fba5c06de40 auth_method 0 2024-03-25T18:03:18.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.537+0000 7fba6cff9640 10 client.4542.objecter ms_handle_connect 0x7fba5c06de40 2024-03-25T18:03:18.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.543+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fba680549a0 con 0x7fba7014be90 2024-03-25T18:03:18.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.543+0000 7fba6cff9640 10 client.4542.objecter ms_dispatch 0x7fba70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:03:18.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.543+0000 7fba6cff9640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:03:18.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.543+0000 7fba6cff9640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:03:18.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.543+0000 7fba6cff9640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-25T18:03:18.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.543+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba680504b0 con 0x7fba7014be90 2024-03-25T18:03:18.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.543+0000 7fba6cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:18.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.543+0000 7fba6cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:18.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.799+0000 7fba76414640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:18.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.799+0000 7fba76414640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:18.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.799+0000 7fba76414640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:03:18.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.799+0000 7fba76414640 1 -- 172.21.15.18:0/3612885797 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba70140af0 con 0x7fba7014be90 2024-03-25T18:03:18.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.831+0000 7fba6cff9640 1 -- 172.21.15.18:0/3612885797 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fba680523e0 con 0x7fba7014be90 2024-03-25T18:03:18.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.831+0000 7fba6cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:18.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.831+0000 7fba6cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:18.833 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:03:18.833 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:03:18.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.833+0000 7fba76414640 10 librados: watch_flush enter 2024-03-25T18:03:18.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.833+0000 7fba76414640 10 librados: watch_flush exit 2024-03-25T18:03:18.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.833+0000 7fba76414640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-25T18:03:18.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.833+0000 7fba76414640 10 client.4542.objecter successfully canceled tick 2024-03-25T18:03:18.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.833+0000 7fba76414640 1 -- 172.21.15.18:0/3612885797 >> v1:172.21.15.18:6800/3721244451 conn(0x7fba5c06de40 legacy=0x7fba5c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:18.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.833+0000 7fba76414640 10 monclient: shutdown 2024-03-25T18:03:18.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.833+0000 7fba76414640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:18.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.833+0000 7fba76414640 1 -- 172.21.15.18:0/3612885797 >> v1:172.21.15.18:6789/0 conn(0x7fba7014be90 legacy=0x7fba701320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:18.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.833+0000 7fba76414640 1 -- 172.21.15.18:0/3612885797 shutdown_connections 2024-03-25T18:03:18.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.833+0000 7fba76414640 1 -- 172.21.15.18:0/3612885797 >> 172.21.15.18:0/3612885797 conn(0x7fba700617a0 msgr2=0x7fba700b63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:18.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.833+0000 7fba76414640 1 -- 172.21.15.18:0/3612885797 shutdown_connections 2024-03-25T18:03:18.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.834+0000 7fba76414640 1 -- 172.21.15.18:0/3612885797 wait complete. 2024-03-25T18:03:18.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:18.834+0000 7fba76414640 1 librados: shutdown 2024-03-25T18:03:18.846 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:03:25.651 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:03:25.768 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.767+0000 7f2b0f596640 10 monclient: get_monmap_and_config 2024-03-25T18:03:25.768 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.767+0000 7f2b0f596640 10 monclient: build_initial_monmap 2024-03-25T18:03:25.768 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.767+0000 7f2b0f596640 10 monclient: monmap: 2024-03-25T18:03:25.768 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:03:25.768 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:25.769 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:03:25.768639+0000 2024-03-25T18:03:25.769 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:03:25.768639+0000 2024-03-25T18:03:25.769 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:25.769 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:25.769 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:03:25.769 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:03:25.769 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:03:25.769 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:25.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.767+0000 7f2b0f596640 1 Processor -- start 2024-03-25T18:03:25.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.768+0000 7f2b0f596640 1 -- start start 2024-03-25T18:03:25.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.768+0000 7f2b0f596640 10 monclient: init 2024-03-25T18:03:25.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.768+0000 7f2b0f596640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:25.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.768+0000 7f2b0f596640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:03:25.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.768+0000 7f2b0f596640 10 monclient(hunting): picked mon.noname-c con 0x7f2b080a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:03:25.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.768+0000 7f2b0f596640 10 monclient(hunting): picked mon.noname-a con 0x7f2b0813f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:03:25.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.768+0000 7f2b0f596640 10 monclient(hunting): picked mon.noname-b con 0x7f2b081430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:03:25.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.768+0000 7f2b0f596640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b08142520 con 0x7f2b0813f710 2024-03-25T18:03:25.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.768+0000 7f2b0f596640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b080b28a0 con 0x7f2b080a98f0 2024-03-25T18:03:25.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.768+0000 7f2b0f596640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b080b2a30 con 0x7f2b081430c0 2024-03-25T18:03:25.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.768+0000 7f2b0f596640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:25.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.768+0000 7f2b0f596640 10 monclient(hunting): authenticate will time out at 7334.347601s 2024-03-25T18:03:25.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.776+0000 7f2b0d332640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f2b080a98f0 0x7f2b080a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:38080/0 (socket says 172.21.15.18:38080) 2024-03-25T18:03:25.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.776+0000 7f2b0d332640 1 -- 172.21.15.18:0/2580435765 learned_addr learned my addr 172.21.15.18:0/2580435765 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:03:25.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.787+0000 7f2afffff640 1 -- 172.21.15.18:0/2580435765 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 616123467 0 0) 0x7f2b080b28a0 con 0x7f2b080a98f0 2024-03-25T18:03:25.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.787+0000 7f2afffff640 10 monclient(hunting): _negotiate 2024-03-25T18:03:25.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.787+0000 7f2afffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:25.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.787+0000 7f2afffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:25.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.787+0000 7f2afffff640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:03:25.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.787+0000 7f2afffff640 1 -- 172.21.15.18:0/2580435765 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af4003520 con 0x7f2b080a98f0 2024-03-25T18:03:25.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.794+0000 7f2afffff640 1 -- 172.21.15.18:0/2580435765 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9102389 0 0) 0x7f2b08142520 con 0x7f2b0813f710 2024-03-25T18:03:25.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.794+0000 7f2afffff640 10 monclient(hunting): _negotiate 2024-03-25T18:03:25.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.794+0000 7f2afffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:25.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.794+0000 7f2afffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:25.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.794+0000 7f2afffff640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:03:25.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.794+0000 7f2afffff640 1 -- 172.21.15.18:0/2580435765 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b080b28a0 con 0x7f2b0813f710 2024-03-25T18:03:25.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.804+0000 7f2afffff640 1 -- 172.21.15.18:0/2580435765 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863410551 0 0) 0x7f2b080b2a30 con 0x7f2b081430c0 2024-03-25T18:03:25.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.804+0000 7f2afffff640 10 monclient(hunting): _negotiate 2024-03-25T18:03:25.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.804+0000 7f2afffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:25.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.804+0000 7f2afffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:25.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.804+0000 7f2afffff640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:03:25.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.804+0000 7f2afffff640 1 -- 172.21.15.18:0/2580435765 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b08142520 con 0x7f2b081430c0 2024-03-25T18:03:25.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.811+0000 7f2afffff640 1 -- 172.21.15.18:0/2580435765 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3470870415 0 0) 0x7f2b080b28a0 con 0x7f2b0813f710 2024-03-25T18:03:25.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.811+0000 7f2afffff640 1 -- 172.21.15.18:0/2580435765 >> v1:172.21.15.18:6790/0 conn(0x7f2b081430c0 legacy=0x7f2b0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:25.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.811+0000 7f2afffff640 1 -- 172.21.15.18:0/2580435765 >> v1:172.21.15.169:6789/0 conn(0x7f2b080a98f0 legacy=0x7f2b080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:25.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.811+0000 7f2afffff640 10 monclient: _finish_hunting 0 2024-03-25T18:03:25.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.811+0000 7f2afffff640 1 monclient: found mon.noname-a 2024-03-25T18:03:25.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.811+0000 7f2afffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:25.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.811+0000 7f2afffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:03:25.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.811+0000 7f2afffff640 1 -- 172.21.15.18:0/2580435765 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b080b44c0 con 0x7f2b0813f710 2024-03-25T18:03:25.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.811+0000 7f2afffff640 10 monclient: _finish_auth 0 2024-03-25T18:03:25.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.811+0000 7f2afffff640 10 monclient: _check_auth_tickets 2024-03-25T18:03:25.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.811+0000 7f2afffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:25.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.811+0000 7f2afffff640 1 -- 172.21.15.18:0/2580435765 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f2af8001670 con 0x7f2b0813f710 2024-03-25T18:03:25.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.811+0000 7f2afffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:25.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.811+0000 7f2afffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:03:25.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.811+0000 7f2afffff640 10 monclient: dump: 2024-03-25T18:03:25.813 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:25.814 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:25.814 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:25.814 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:25.814 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:25.814 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:25.814 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:25.814 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:25.814 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:25.814 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:25.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.811+0000 7f2b0f596640 5 monclient: authenticate success, global_id 4545 2024-03-25T18:03:25.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.811+0000 7f2b0f596640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:25.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.828+0000 7f2afffff640 1 -- 172.21.15.18:0/2580435765 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2af80017d0 con 0x7f2b0813f710 2024-03-25T18:03:25.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.828+0000 7f2afffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:25.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.828+0000 7f2afffff640 1 -- 172.21.15.18:0/2580435765 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f2af8001be0 con 0x7f2b0813f710 2024-03-25T18:03:25.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.828+0000 7f2b0f596640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:25.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.828+0000 7f2b0f596640 4 set_mon_vals no callback set 2024-03-25T18:03:25.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.828+0000 7f2afffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:25.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.828+0000 7f2afffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:25.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.828+0000 7f2afffff640 10 monclient: dump: 2024-03-25T18:03:25.830 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:25.830 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:25.830 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:25.830 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:25.830 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:25.830 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:25.830 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:25.830 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:25.830 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:25.830 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:25.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.829+0000 7f2b0f596640 10 monclient: shutdown 2024-03-25T18:03:25.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.829+0000 7f2b0f596640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:25.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.829+0000 7f2b0f596640 1 -- 172.21.15.18:0/2580435765 >> v1:172.21.15.18:6789/0 conn(0x7f2b0813f710 legacy=0x7f2b081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:25.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.829+0000 7f2b0f596640 1 -- 172.21.15.18:0/2580435765 shutdown_connections 2024-03-25T18:03:25.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.829+0000 7f2b0f596640 1 -- 172.21.15.18:0/2580435765 >> 172.21.15.18:0/2580435765 conn(0x7f2b080b7ea0 msgr2=0x7f2b080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:25.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.829+0000 7f2b0f596640 1 -- 172.21.15.18:0/2580435765 shutdown_connections 2024-03-25T18:03:25.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.829+0000 7f2b0f596640 1 -- 172.21.15.18:0/2580435765 wait complete. 2024-03-25T18:03:25.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.830+0000 7f2b0f596640 10 monclient: build_initial_monmap 2024-03-25T18:03:25.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.830+0000 7f2b0f596640 10 monclient: monmap: 2024-03-25T18:03:25.833 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:03:25.833 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:25.833 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:03:25.833 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:03:25.833 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:25.833 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:03:25.833 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:03:25.833 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:03:25.833 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:03:25.833 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:25.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.830+0000 7f2b0f596640 1 librados: starting msgr at 2024-03-25T18:03:25.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.830+0000 7f2b0f596640 1 librados: starting objecter 2024-03-25T18:03:25.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.830+0000 7f2b0f596640 1 Processor -- start 2024-03-25T18:03:25.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.830+0000 7f2b0f596640 1 -- start start 2024-03-25T18:03:25.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.830+0000 7f2b0f596640 1 librados: setting wanted keys 2024-03-25T18:03:25.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.830+0000 7f2b0f596640 1 librados: calling monclient init 2024-03-25T18:03:25.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.830+0000 7f2b0f596640 10 monclient: init 2024-03-25T18:03:25.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.831+0000 7f2b0f596640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:25.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.831+0000 7f2b0f596640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:03:25.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.831+0000 7f2b0f596640 10 monclient(hunting): picked mon.noname-b con 0x7f2b080a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:03:25.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.831+0000 7f2b0f596640 10 monclient(hunting): picked mon.noname-c con 0x7f2b0813f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:03:25.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.831+0000 7f2b0f596640 10 monclient(hunting): picked mon.noname-a con 0x7f2b081430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:03:25.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.831+0000 7f2b0f596640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b08139190 con 0x7f2b081430c0 2024-03-25T18:03:25.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.831+0000 7f2b0f596640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b08137590 con 0x7f2b080a98f0 2024-03-25T18:03:25.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.831+0000 7f2b0f596640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b08138b40 con 0x7f2b0813f710 2024-03-25T18:03:25.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.831+0000 7f2b0f596640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:25.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.831+0000 7f2b0f596640 10 monclient(hunting): authenticate will time out at 7334.409978s 2024-03-25T18:03:25.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.839+0000 7f2b0d332640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f2b080a98f0 0x7f2b08136e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:38108/0 (socket says 172.21.15.18:38108) 2024-03-25T18:03:25.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.839+0000 7f2b0d332640 1 -- 172.21.15.18:0/2850086172 learned_addr learned my addr 172.21.15.18:0/2850086172 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:03:25.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.842+0000 7f2afdffb640 10 client.?.objecter ms_handle_connect 0x7f2b080a98f0 2024-03-25T18:03:25.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.842+0000 7f2afdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:25.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.848+0000 7f2afdffb640 10 client.?.objecter ms_handle_connect 0x7f2b0813f710 2024-03-25T18:03:25.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.848+0000 7f2afdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:25.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.852+0000 7f2afdffb640 10 client.?.objecter ms_handle_connect 0x7f2b081430c0 2024-03-25T18:03:25.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.852+0000 7f2afdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:25.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.853+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559987450 0 0) 0x7f2b08138b40 con 0x7f2b0813f710 2024-03-25T18:03:25.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.853+0000 7f2afdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:03:25.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.853+0000 7f2afdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:25.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.853+0000 7f2afdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:25.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.853+0000 7f2afdffb640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:03:25.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.853+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aec004eb0 con 0x7f2b0813f710 2024-03-25T18:03:25.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.858+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199013071 0 0) 0x7f2b08137590 con 0x7f2b080a98f0 2024-03-25T18:03:25.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.858+0000 7f2afdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:03:25.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.858+0000 7f2afdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:25.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.858+0000 7f2afdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:25.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.858+0000 7f2afdffb640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:03:25.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.858+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b08138b40 con 0x7f2b080a98f0 2024-03-25T18:03:25.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.870+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999641413 0 0) 0x7f2b08139190 con 0x7f2b081430c0 2024-03-25T18:03:25.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.870+0000 7f2afdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:03:25.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.870+0000 7f2afdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:25.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.870+0000 7f2afdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:25.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.870+0000 7f2afdffb640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:03:25.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.870+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b08137590 con 0x7f2b081430c0 2024-03-25T18:03:25.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.879+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 736523397 0 0) 0x7f2aec004eb0 con 0x7f2b0813f710 2024-03-25T18:03:25.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.879+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b08139190 con 0x7f2b0813f710 2024-03-25T18:03:25.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.879+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f2af8001ba0 con 0x7f2b0813f710 2024-03-25T18:03:25.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.879+0000 7f2afdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:25.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.879+0000 7f2afdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:25.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.879+0000 7f2afdffb640 10 monclient(hunting): dump: 2024-03-25T18:03:25.881 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:25.881 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:25.881 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:25.882 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:25.882 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:25.882 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:25.882 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:25.882 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:25.882 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:25.882 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:25.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.879+0000 7f2afdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:25.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.880+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1660872322 0 0) 0x7f2b08138b40 con 0x7f2b080a98f0 2024-03-25T18:03:25.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.880+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2aec004eb0 con 0x7f2b080a98f0 2024-03-25T18:03:25.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.880+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f2b040018d0 con 0x7f2b080a98f0 2024-03-25T18:03:25.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.880+0000 7f2afdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:25.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.880+0000 7f2afdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:25.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.880+0000 7f2afdffb640 10 monclient(hunting): dump: 2024-03-25T18:03:25.882 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:25.883 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:25.883 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:25.883 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:25.883 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:25.883 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:25.883 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:25.883 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:25.883 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:25.883 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:25.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3377677273 0 0) 0x7f2b08139190 con 0x7f2b0813f710 2024-03-25T18:03:25.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 >> v1:172.21.15.169:6789/0 conn(0x7f2b080a98f0 legacy=0x7f2b08136e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:25.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 >> v1:172.21.15.18:6789/0 conn(0x7f2b081430c0 legacy=0x7f2b081e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:25.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2afdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:03:25.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2afdffb640 1 monclient: found mon.c 2024-03-25T18:03:25.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2afdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:25.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2afdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:25.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b08136bf0 con 0x7f2b0813f710 2024-03-25T18:03:25.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2afdffb640 10 monclient: _finish_auth 0 2024-03-25T18:03:25.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2afdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:03:25.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2afdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:25.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2b0f596640 5 monclient: authenticate success, global_id 4556 2024-03-25T18:03:25.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2b0f596640 10 monclient: _renew_subs 2024-03-25T18:03:25.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2b0f596640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:25.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2b0f596640 1 -- 172.21.15.18:0/2850086172 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b0813afe0 con 0x7f2b0813f710 2024-03-25T18:03:25.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2b0f596640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:25.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2b0f596640 10 monclient: _renew_subs 2024-03-25T18:03:25.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2b0f596640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:25.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2b0f596640 1 -- 172.21.15.18:0/2850086172 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b08139620 con 0x7f2b0813f710 2024-03-25T18:03:25.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.890+0000 7f2b0f596640 1 librados: init done 2024-03-25T18:03:25.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.891+0000 7f2b0f596640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:25.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.891+0000 7f2b0f596640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:25.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.891+0000 7f2b0f596640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:25.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:25.891+0000 7f2b0f596640 1 -- 172.21.15.18:0/2850086172 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b08138f60 con 0x7f2b0813f710 2024-03-25T18:03:26.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.194+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2af80038a0 con 0x7f2b0813f710 2024-03-25T18:03:26.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.194+0000 7f2afdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:26.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.194+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f2af8003bb0 con 0x7f2b0813f710 2024-03-25T18:03:26.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.194+0000 7f2afdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:26.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.194+0000 7f2afdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:26.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.194+0000 7f2afdffb640 10 monclient: dump: 2024-03-25T18:03:26.196 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:26.196 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:26.196 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:26.196 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:26.196 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:26.196 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:26.196 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:26.196 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:26.196 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:26.196 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:26.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.194+0000 7f2aff7fe640 4 set_mon_vals no callback set 2024-03-25T18:03:26.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.195+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f2af8019b80 con 0x7f2b0813f710 2024-03-25T18:03:26.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.195+0000 7f2b0d332640 10 monclient: get_auth_request con 0x7f2aec06df70 auth_method 0 2024-03-25T18:03:26.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.196+0000 7f2afdffb640 10 client.4556.objecter ms_handle_connect 0x7f2aec06df70 2024-03-25T18:03:26.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.204+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f2af8053720 con 0x7f2b0813f710 2024-03-25T18:03:26.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.204+0000 7f2afdffb640 10 client.4556.objecter ms_dispatch 0x7f2b08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:03:26.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.204+0000 7f2afdffb640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:03:26.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.204+0000 7f2afdffb640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:03:26.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.204+0000 7f2afdffb640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-25T18:03:26.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.204+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2af804f230 con 0x7f2b0813f710 2024-03-25T18:03:26.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.204+0000 7f2afdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:26.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.204+0000 7f2afdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:26.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.458+0000 7f2b0f596640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:26.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.458+0000 7f2b0f596640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:26.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.458+0000 7f2b0f596640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:26.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.458+0000 7f2b0f596640 1 -- 172.21.15.18:0/2850086172 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b08139340 con 0x7f2b0813f710 2024-03-25T18:03:26.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.485+0000 7f2afdffb640 1 -- 172.21.15.18:0/2850086172 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f2b08139340 con 0x7f2b0813f710 2024-03-25T18:03:26.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.485+0000 7f2afdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:26.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.485+0000 7f2afdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:26.487 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:03:26.487 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:03:26.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.488+0000 7f2b0f596640 10 librados: watch_flush enter 2024-03-25T18:03:26.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.488+0000 7f2b0f596640 10 librados: watch_flush exit 2024-03-25T18:03:26.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.488+0000 7f2b0f596640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-25T18:03:26.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.488+0000 7f2b0f596640 10 client.4556.objecter successfully canceled tick 2024-03-25T18:03:26.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.488+0000 7f2b0f596640 1 -- 172.21.15.18:0/2850086172 >> v1:172.21.15.18:6800/3721244451 conn(0x7f2aec06df70 legacy=0x7f2aec070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:26.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.488+0000 7f2b0f596640 10 monclient: shutdown 2024-03-25T18:03:26.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.488+0000 7f2b0f596640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:26.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.488+0000 7f2b0f596640 1 -- 172.21.15.18:0/2850086172 >> v1:172.21.15.18:6790/0 conn(0x7f2b0813f710 legacy=0x7f2b081364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:26.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.488+0000 7f2b0f596640 1 -- 172.21.15.18:0/2850086172 shutdown_connections 2024-03-25T18:03:26.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.488+0000 7f2b0f596640 1 -- 172.21.15.18:0/2850086172 >> 172.21.15.18:0/2850086172 conn(0x7f2b080b7ea0 msgr2=0x7f2b08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:26.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.488+0000 7f2b0f596640 1 -- 172.21.15.18:0/2850086172 shutdown_connections 2024-03-25T18:03:26.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.488+0000 7f2b0f596640 1 -- 172.21.15.18:0/2850086172 wait complete. 2024-03-25T18:03:26.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:26.488+0000 7f2b0f596640 1 librados: shutdown 2024-03-25T18:03:26.501 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:03:28.343 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:26.136 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:28.355 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:26.148 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:28.394 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:26.187 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:28.401 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:26.149 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:28.405 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:26.154 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:28.407 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:26.199 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:28.413 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:26.205 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:28.432 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:26.181 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:28.435 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:26.228 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:33.306 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:03:33.412 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.411+0000 7f26e8ef1640 10 monclient: get_monmap_and_config 2024-03-25T18:03:33.412 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.411+0000 7f26e8ef1640 10 monclient: build_initial_monmap 2024-03-25T18:03:33.412 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.411+0000 7f26e8ef1640 10 monclient: monmap: 2024-03-25T18:03:33.412 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:03:33.412 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:33.412 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:03:33.412487+0000 2024-03-25T18:03:33.413 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:03:33.412487+0000 2024-03-25T18:03:33.413 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:33.413 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:33.413 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:03:33.413 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:03:33.413 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:03:33.413 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:33.414 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.411+0000 7f26e8ef1640 1 Processor -- start 2024-03-25T18:03:33.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.411+0000 7f26e8ef1640 1 -- start start 2024-03-25T18:03:33.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.411+0000 7f26e8ef1640 10 monclient: init 2024-03-25T18:03:33.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.412+0000 7f26e8ef1640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:33.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.412+0000 7f26e8ef1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:03:33.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.412+0000 7f26e8ef1640 10 monclient(hunting): picked mon.noname-a con 0x7f26e40b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T18:03:33.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.412+0000 7f26e8ef1640 10 monclient(hunting): picked mon.noname-c con 0x7f26e40b1730 addr v1:172.21.15.169:6789/0 2024-03-25T18:03:33.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.412+0000 7f26e8ef1640 10 monclient(hunting): picked mon.noname-b con 0x7f26e414be90 addr v1:172.21.15.18:6790/0 2024-03-25T18:03:33.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.412+0000 7f26e8ef1640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e40af940 con 0x7f26e40b4d60 2024-03-25T18:03:33.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.412+0000 7f26e8ef1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e40b0a70 con 0x7f26e40b1730 2024-03-25T18:03:33.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.412+0000 7f26e8ef1640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e40b0d50 con 0x7f26e414be90 2024-03-25T18:03:33.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.412+0000 7f26e8ef1640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:33.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.412+0000 7f26e8ef1640 10 monclient(hunting): authenticate will time out at 7341.991292s 2024-03-25T18:03:33.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.416+0000 7f26dbfff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f26e40b1730 0x7f26e40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:42496/0 (socket says 172.21.15.18:42496) 2024-03-25T18:03:33.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.416+0000 7f26dbfff640 1 -- 172.21.15.18:0/2576042464 learned_addr learned my addr 172.21.15.18:0/2576042464 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:03:33.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.425+0000 7f26e1d9b640 1 -- 172.21.15.18:0/2576042464 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599619565 0 0) 0x7f26e40b0a70 con 0x7f26e40b1730 2024-03-25T18:03:33.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.425+0000 7f26e1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:03:33.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.425+0000 7f26e1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:33.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.425+0000 7f26e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:33.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.425+0000 7f26e1d9b640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:03:33.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.425+0000 7f26e1d9b640 1 -- 172.21.15.18:0/2576042464 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26d0003520 con 0x7f26e40b1730 2024-03-25T18:03:33.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.430+0000 7f26e1d9b640 1 -- 172.21.15.18:0/2576042464 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809088602 0 0) 0x7f26e40b0d50 con 0x7f26e414be90 2024-03-25T18:03:33.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.430+0000 7f26e1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:03:33.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.430+0000 7f26e1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:33.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.430+0000 7f26e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:33.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.430+0000 7f26e1d9b640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:03:33.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.430+0000 7f26e1d9b640 1 -- 172.21.15.18:0/2576042464 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e40b0a70 con 0x7f26e414be90 2024-03-25T18:03:33.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.435+0000 7f26e1d9b640 1 -- 172.21.15.18:0/2576042464 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634976159 0 0) 0x7f26e40af940 con 0x7f26e40b4d60 2024-03-25T18:03:33.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.435+0000 7f26e1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:03:33.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.435+0000 7f26e1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:33.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.435+0000 7f26e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:33.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.435+0000 7f26e1d9b640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:03:33.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.435+0000 7f26e1d9b640 1 -- 172.21.15.18:0/2576042464 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e40b0d50 con 0x7f26e40b4d60 2024-03-25T18:03:33.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e1d9b640 1 -- 172.21.15.18:0/2576042464 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 981459397 0 0) 0x7f26d0003520 con 0x7f26e40b1730 2024-03-25T18:03:33.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e1d9b640 1 -- 172.21.15.18:0/2576042464 >> v1:172.21.15.18:6790/0 conn(0x7f26e414be90 legacy=0x7f26e414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:33.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e1d9b640 1 -- 172.21.15.18:0/2576042464 >> v1:172.21.15.18:6789/0 conn(0x7f26e40b4d60 legacy=0x7f26e413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:33.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:03:33.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e1d9b640 1 monclient: found mon.noname-c 2024-03-25T18:03:33.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:33.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:03:33.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e1d9b640 1 -- 172.21.15.18:0/2576042464 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e40b1030 con 0x7f26e40b1730 2024-03-25T18:03:33.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:03:33.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:03:33.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:33.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e1d9b640 1 -- 172.21.15.18:0/2576042464 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f26cc001690 con 0x7f26e40b1730 2024-03-25T18:03:33.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e8ef1640 5 monclient: authenticate success, global_id 4537 2024-03-25T18:03:33.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e8ef1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:33.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:33.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:33.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e1d9b640 10 monclient: dump: 2024-03-25T18:03:33.443 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:33.443 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:33.443 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:33.443 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:33.443 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:33.443 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:33.443 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:33.443 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:33.443 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:33.443 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:33.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.440+0000 7f26e8ef1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:33.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.459+0000 7f26e1d9b640 1 -- 172.21.15.18:0/2576042464 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26cc0017f0 con 0x7f26e40b1730 2024-03-25T18:03:33.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.459+0000 7f26e1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:33.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.459+0000 7f26e1d9b640 1 -- 172.21.15.18:0/2576042464 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f26cc001c00 con 0x7f26e40b1730 2024-03-25T18:03:33.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.459+0000 7f26e8ef1640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:33.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.459+0000 7f26e8ef1640 4 set_mon_vals no callback set 2024-03-25T18:03:33.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.459+0000 7f26e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:33.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.459+0000 7f26e1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:33.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.459+0000 7f26e1d9b640 10 monclient: dump: 2024-03-25T18:03:33.461 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:33.461 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:33.461 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:33.461 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:33.461 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:33.461 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:33.461 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:33.461 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:33.461 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:33.461 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:33.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.460+0000 7f26e8ef1640 10 monclient: shutdown 2024-03-25T18:03:33.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.460+0000 7f26e8ef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:33.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.460+0000 7f26e8ef1640 1 -- 172.21.15.18:0/2576042464 >> v1:172.21.15.169:6789/0 conn(0x7f26e40b1730 legacy=0x7f26e40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:33.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.460+0000 7f26e8ef1640 1 -- 172.21.15.18:0/2576042464 shutdown_connections 2024-03-25T18:03:33.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.460+0000 7f26e8ef1640 1 -- 172.21.15.18:0/2576042464 >> 172.21.15.18:0/2576042464 conn(0x7f26e40617a0 msgr2=0x7f26e4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:33.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.460+0000 7f26e8ef1640 1 -- 172.21.15.18:0/2576042464 shutdown_connections 2024-03-25T18:03:33.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.460+0000 7f26e8ef1640 1 -- 172.21.15.18:0/2576042464 wait complete. 2024-03-25T18:03:33.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.461+0000 7f26e8ef1640 10 monclient: build_initial_monmap 2024-03-25T18:03:33.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.461+0000 7f26e8ef1640 10 monclient: monmap: 2024-03-25T18:03:33.463 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:03:33.463 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:33.464 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:03:33.464 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:03:33.464 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:33.464 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:03:33.464 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:03:33.464 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:03:33.464 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:03:33.464 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:33.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.461+0000 7f26e8ef1640 1 librados: starting msgr at 2024-03-25T18:03:33.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.461+0000 7f26e8ef1640 1 librados: starting objecter 2024-03-25T18:03:33.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.461+0000 7f26e8ef1640 1 Processor -- start 2024-03-25T18:03:33.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.461+0000 7f26e8ef1640 1 -- start start 2024-03-25T18:03:33.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.461+0000 7f26e8ef1640 1 librados: setting wanted keys 2024-03-25T18:03:33.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.461+0000 7f26e8ef1640 1 librados: calling monclient init 2024-03-25T18:03:33.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.461+0000 7f26e8ef1640 10 monclient: init 2024-03-25T18:03:33.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.462+0000 7f26e8ef1640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:33.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.462+0000 7f26e8ef1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:03:33.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.462+0000 7f26e8ef1640 10 monclient(hunting): picked mon.noname-b con 0x7f26e40b1730 addr v1:172.21.15.169:6789/0 2024-03-25T18:03:33.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.462+0000 7f26e8ef1640 10 monclient(hunting): picked mon.noname-c con 0x7f26e40b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T18:03:33.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.462+0000 7f26e8ef1640 10 monclient(hunting): picked mon.noname-a con 0x7f26e414be90 addr v1:172.21.15.18:6789/0 2024-03-25T18:03:33.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.462+0000 7f26e8ef1640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e41dac30 con 0x7f26e414be90 2024-03-25T18:03:33.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.462+0000 7f26e8ef1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e4142560 con 0x7f26e40b1730 2024-03-25T18:03:33.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.462+0000 7f26e8ef1640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e41db860 con 0x7f26e40b4d60 2024-03-25T18:03:33.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.462+0000 7f26e8ef1640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:33.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.463+0000 7f26e8ef1640 10 monclient(hunting): authenticate will time out at 7342.041668s 2024-03-25T18:03:33.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.474+0000 7f26e2d9d640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f26e414be90 0x7f26e4141750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:33540/0 (socket says 172.21.15.18:33540) 2024-03-25T18:03:33.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.474+0000 7f26e2d9d640 1 -- 172.21.15.18:0/1523790040 learned_addr learned my addr 172.21.15.18:0/1523790040 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:03:33.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.475+0000 7f26daffd640 10 client.?.objecter ms_handle_connect 0x7f26e414be90 2024-03-25T18:03:33.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.475+0000 7f26daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:33.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.477+0000 7f26daffd640 10 client.?.objecter ms_handle_connect 0x7f26e40b4d60 2024-03-25T18:03:33.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.477+0000 7f26daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:33.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.487+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221584821 0 0) 0x7f26e41db860 con 0x7f26e40b4d60 2024-03-25T18:03:33.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.487+0000 7f26daffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:33.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.487+0000 7f26daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:33.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.487+0000 7f26daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:33.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.487+0000 7f26daffd640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:03:33.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.487+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c4004eb0 con 0x7f26e40b4d60 2024-03-25T18:03:33.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.495+0000 7f26daffd640 10 client.?.objecter ms_handle_connect 0x7f26e40b1730 2024-03-25T18:03:33.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.495+0000 7f26daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:33.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.496+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473450761 0 0) 0x7f26e41dac30 con 0x7f26e414be90 2024-03-25T18:03:33.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.496+0000 7f26daffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:33.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.496+0000 7f26daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:33.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.496+0000 7f26daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:33.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.496+0000 7f26daffd640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:03:33.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.496+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e41db860 con 0x7f26e414be90 2024-03-25T18:03:33.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.504+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2528515522 0 0) 0x7f26c4004eb0 con 0x7f26e40b4d60 2024-03-25T18:03:33.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.504+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26e41dac30 con 0x7f26e40b4d60 2024-03-25T18:03:33.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.504+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f26cc001b00 con 0x7f26e40b4d60 2024-03-25T18:03:33.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.504+0000 7f26daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:33.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.504+0000 7f26daffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:33.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.504+0000 7f26daffd640 10 monclient(hunting): dump: 2024-03-25T18:03:33.506 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:33.506 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:33.506 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:33.506 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:33.506 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:33.506 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:33.506 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:33.506 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:33.507 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:33.507 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:33.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.504+0000 7f26daffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:33.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.507+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345993306 0 0) 0x7f26e4142560 con 0x7f26e40b1730 2024-03-25T18:03:33.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.507+0000 7f26daffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:33.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.507+0000 7f26daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:33.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.507+0000 7f26daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:33.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.507+0000 7f26daffd640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:03:33.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.507+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c4004eb0 con 0x7f26e40b1730 2024-03-25T18:03:33.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2249851765 0 0) 0x7f26e41db860 con 0x7f26e414be90 2024-03-25T18:03:33.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26e4142560 con 0x7f26e414be90 2024-03-25T18:03:33.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f26dc001bf0 con 0x7f26e414be90 2024-03-25T18:03:33.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:33.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26daffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:33.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26daffd640 10 monclient(hunting): dump: 2024-03-25T18:03:33.519 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:33.520 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:33.520 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:33.520 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:33.520 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:33.520 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:33.520 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:33.520 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:33.520 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:33.520 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:33.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 294462782 0 0) 0x7f26e41dac30 con 0x7f26e40b4d60 2024-03-25T18:03:33.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 >> v1:172.21.15.169:6789/0 conn(0x7f26e40b1730 legacy=0x7f26e4135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:33.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 >> v1:172.21.15.18:6789/0 conn(0x7f26e414be90 legacy=0x7f26e4141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:33.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26daffd640 10 monclient: _finish_hunting 0 2024-03-25T18:03:33.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26daffd640 1 monclient: found mon.c 2024-03-25T18:03:33.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:33.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26daffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:33.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e41db9f0 con 0x7f26e40b4d60 2024-03-25T18:03:33.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26daffd640 10 monclient: _finish_auth 0 2024-03-25T18:03:33.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26daffd640 10 monclient: _check_auth_tickets 2024-03-25T18:03:33.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:33.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26e8ef1640 5 monclient: authenticate success, global_id 4562 2024-03-25T18:03:33.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26e8ef1640 10 monclient: _renew_subs 2024-03-25T18:03:33.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26e8ef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:33.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26e8ef1640 1 -- 172.21.15.18:0/1523790040 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e4141e60 con 0x7f26e40b4d60 2024-03-25T18:03:33.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26e8ef1640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:33.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26e8ef1640 10 monclient: _renew_subs 2024-03-25T18:03:33.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26e8ef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:33.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26e8ef1640 1 -- 172.21.15.18:0/1523790040 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e41377d0 con 0x7f26e40b4d60 2024-03-25T18:03:33.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.517+0000 7f26e8ef1640 1 librados: init done 2024-03-25T18:03:33.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.518+0000 7f26e8ef1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:33.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.518+0000 7f26e8ef1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:33.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.518+0000 7f26e8ef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:33.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.518+0000 7f26e8ef1640 1 -- 172.21.15.18:0/1523790040 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e41daa90 con 0x7f26e40b4d60 2024-03-25T18:03:33.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.832+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26cc001220 con 0x7f26e40b4d60 2024-03-25T18:03:33.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.832+0000 7f26daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:33.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.832+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f26cc004a90 con 0x7f26e40b4d60 2024-03-25T18:03:33.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.832+0000 7f26daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:33.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.832+0000 7f26daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:33.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.832+0000 7f26daffd640 10 monclient: dump: 2024-03-25T18:03:33.834 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:33.834 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:33.834 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:33.834 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:33.834 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:33.834 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:33.834 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:33.835 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:33.835 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:33.835 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:33.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.832+0000 7f26e159a640 4 set_mon_vals no callback set 2024-03-25T18:03:33.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.833+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f26cc01aa60 con 0x7f26e40b4d60 2024-03-25T18:03:33.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.834+0000 7f26e259c640 10 monclient: get_auth_request con 0x7f26c406dfb0 auth_method 0 2024-03-25T18:03:33.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.834+0000 7f26daffd640 10 client.4562.objecter ms_handle_connect 0x7f26c406dfb0 2024-03-25T18:03:33.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.839+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f26cc0544c0 con 0x7f26e40b4d60 2024-03-25T18:03:33.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.839+0000 7f26daffd640 10 client.4562.objecter ms_dispatch 0x7f26e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:03:33.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.839+0000 7f26daffd640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:03:33.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.839+0000 7f26daffd640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:03:33.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.839+0000 7f26daffd640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-25T18:03:33.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.839+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f26cc04ffd0 con 0x7f26e40b4d60 2024-03-25T18:03:33.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.840+0000 7f26daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:33.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:33.840+0000 7f26daffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:34.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.095+0000 7f26e8ef1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:34.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.095+0000 7f26e8ef1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:34.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.095+0000 7f26e8ef1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:34.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.095+0000 7f26e8ef1640 1 -- 172.21.15.18:0/1523790040 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26e41db9f0 con 0x7f26e40b4d60 2024-03-25T18:03:34.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.131+0000 7f26daffd640 1 -- 172.21.15.18:0/1523790040 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f26e41db9f0 con 0x7f26e40b4d60 2024-03-25T18:03:34.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.131+0000 7f26daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:34.132 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.131+0000 7f26daffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:34.132 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:03:34.132 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:03:34.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.133+0000 7f26e8ef1640 10 librados: watch_flush enter 2024-03-25T18:03:34.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.133+0000 7f26e8ef1640 10 librados: watch_flush exit 2024-03-25T18:03:34.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.133+0000 7f26e8ef1640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-25T18:03:34.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.133+0000 7f26e8ef1640 10 client.4562.objecter successfully canceled tick 2024-03-25T18:03:34.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.133+0000 7f26e8ef1640 1 -- 172.21.15.18:0/1523790040 >> v1:172.21.15.18:6800/3721244451 conn(0x7f26c406dfb0 legacy=0x7f26c4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:34.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.133+0000 7f26e8ef1640 10 monclient: shutdown 2024-03-25T18:03:34.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.133+0000 7f26e8ef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:34.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.133+0000 7f26e8ef1640 1 -- 172.21.15.18:0/1523790040 >> v1:172.21.15.18:6790/0 conn(0x7f26e40b4d60 legacy=0x7f26e41db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:34.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.133+0000 7f26e8ef1640 1 -- 172.21.15.18:0/1523790040 shutdown_connections 2024-03-25T18:03:34.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.133+0000 7f26e8ef1640 1 -- 172.21.15.18:0/1523790040 >> 172.21.15.18:0/1523790040 conn(0x7f26e40617a0 msgr2=0x7f26e4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:34.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.133+0000 7f26e8ef1640 1 -- 172.21.15.18:0/1523790040 shutdown_connections 2024-03-25T18:03:34.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.133+0000 7f26e8ef1640 1 -- 172.21.15.18:0/1523790040 wait complete. 2024-03-25T18:03:34.136 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:34.133+0000 7f26e8ef1640 1 librados: shutdown 2024-03-25T18:03:34.148 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:03:40.956 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:03:41.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.069+0000 7fbf78715640 10 monclient: get_monmap_and_config 2024-03-25T18:03:41.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.069+0000 7fbf78715640 10 monclient: build_initial_monmap 2024-03-25T18:03:41.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.069+0000 7fbf78715640 10 monclient: monmap: 2024-03-25T18:03:41.071 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:03:41.071 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:41.071 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:03:41.070546+0000 2024-03-25T18:03:41.071 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:03:41.070546+0000 2024-03-25T18:03:41.071 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:41.072 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:41.072 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:03:41.072 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:03:41.072 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:03:41.072 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:41.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.069+0000 7fbf78715640 1 Processor -- start 2024-03-25T18:03:41.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.069+0000 7fbf78715640 1 -- start start 2024-03-25T18:03:41.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.069+0000 7fbf78715640 10 monclient: init 2024-03-25T18:03:41.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.070+0000 7fbf78715640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:41.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.070+0000 7fbf78715640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:03:41.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.070+0000 7fbf78715640 10 monclient(hunting): picked mon.noname-c con 0x7fbf700b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T18:03:41.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.070+0000 7fbf78715640 10 monclient(hunting): picked mon.noname-b con 0x7fbf700b1730 addr v1:172.21.15.18:6790/0 2024-03-25T18:03:41.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.070+0000 7fbf78715640 10 monclient(hunting): picked mon.noname-a con 0x7fbf7014be90 addr v1:172.21.15.18:6789/0 2024-03-25T18:03:41.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.070+0000 7fbf78715640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf700af940 con 0x7fbf7014be90 2024-03-25T18:03:41.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.070+0000 7fbf78715640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf700b0a70 con 0x7fbf700b4d60 2024-03-25T18:03:41.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.070+0000 7fbf78715640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf700b0d50 con 0x7fbf700b1730 2024-03-25T18:03:41.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.070+0000 7fbf78715640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:41.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.070+0000 7fbf78715640 10 monclient(hunting): authenticate will time out at 7349.649298s 2024-03-25T18:03:41.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.071+0000 7fbf764b1640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fbf700b4d60 0x7fbf7013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:56316/0 (socket says 172.21.15.18:56316) 2024-03-25T18:03:41.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.071+0000 7fbf764b1640 1 -- 172.21.15.18:0/2995868139 learned_addr learned my addr 172.21.15.18:0/2995868139 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:03:41.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.142+0000 7fbf754af640 1 -- 172.21.15.18:0/2995868139 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742771022 0 0) 0x7fbf700af940 con 0x7fbf7014be90 2024-03-25T18:03:41.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.142+0000 7fbf754af640 10 monclient(hunting): _negotiate 2024-03-25T18:03:41.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.142+0000 7fbf754af640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:41.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.142+0000 7fbf754af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:41.144 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.142+0000 7fbf754af640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:03:41.144 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.142+0000 7fbf754af640 1 -- 172.21.15.18:0/2995868139 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c003520 con 0x7fbf7014be90 2024-03-25T18:03:41.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.151+0000 7fbf754af640 1 -- 172.21.15.18:0/2995868139 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3283594603 0 0) 0x7fbf5c003520 con 0x7fbf7014be90 2024-03-25T18:03:41.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.151+0000 7fbf754af640 1 -- 172.21.15.18:0/2995868139 >> v1:172.21.15.18:6790/0 conn(0x7fbf700b1730 legacy=0x7fbf700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:41.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.151+0000 7fbf754af640 1 -- 172.21.15.18:0/2995868139 >> v1:172.21.15.169:6789/0 conn(0x7fbf700b4d60 legacy=0x7fbf7013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:41.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.151+0000 7fbf754af640 10 monclient: _finish_hunting 0 2024-03-25T18:03:41.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.151+0000 7fbf754af640 1 monclient: found mon.noname-a 2024-03-25T18:03:41.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.151+0000 7fbf754af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:41.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.151+0000 7fbf754af640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:03:41.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.151+0000 7fbf754af640 1 -- 172.21.15.18:0/2995868139 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf700b1030 con 0x7fbf7014be90 2024-03-25T18:03:41.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.151+0000 7fbf754af640 10 monclient: _finish_auth 0 2024-03-25T18:03:41.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.151+0000 7fbf754af640 10 monclient: _check_auth_tickets 2024-03-25T18:03:41.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.151+0000 7fbf754af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:41.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.151+0000 7fbf754af640 1 -- 172.21.15.18:0/2995868139 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fbf68001a10 con 0x7fbf7014be90 2024-03-25T18:03:41.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.151+0000 7fbf754af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:41.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.151+0000 7fbf754af640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:03:41.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.151+0000 7fbf754af640 10 monclient: dump: 2024-03-25T18:03:41.154 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:41.155 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:41.155 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:41.155 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:41.155 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:41.155 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:41.155 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:41.155 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:41.155 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:41.155 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:41.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.151+0000 7fbf78715640 5 monclient: authenticate success, global_id 4557 2024-03-25T18:03:41.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.151+0000 7fbf78715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:41.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.174+0000 7fbf754af640 1 -- 172.21.15.18:0/2995868139 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf68001b70 con 0x7fbf7014be90 2024-03-25T18:03:41.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.174+0000 7fbf754af640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:41.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.174+0000 7fbf754af640 1 -- 172.21.15.18:0/2995868139 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fbf68001fc0 con 0x7fbf7014be90 2024-03-25T18:03:41.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.174+0000 7fbf78715640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:41.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.174+0000 7fbf78715640 4 set_mon_vals no callback set 2024-03-25T18:03:41.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.174+0000 7fbf754af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:41.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.174+0000 7fbf754af640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:41.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.174+0000 7fbf754af640 10 monclient: dump: 2024-03-25T18:03:41.177 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:41.177 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:41.177 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:41.177 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:41.177 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:41.177 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:41.177 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:41.177 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:41.177 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:41.178 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:41.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.175+0000 7fbf78715640 10 monclient: shutdown 2024-03-25T18:03:41.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.175+0000 7fbf78715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:41.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.175+0000 7fbf78715640 1 -- 172.21.15.18:0/2995868139 >> v1:172.21.15.18:6789/0 conn(0x7fbf7014be90 legacy=0x7fbf7014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:41.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.175+0000 7fbf78715640 1 -- 172.21.15.18:0/2995868139 shutdown_connections 2024-03-25T18:03:41.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.175+0000 7fbf78715640 1 -- 172.21.15.18:0/2995868139 >> 172.21.15.18:0/2995868139 conn(0x7fbf700617a0 msgr2=0x7fbf70061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:41.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.175+0000 7fbf78715640 1 -- 172.21.15.18:0/2995868139 shutdown_connections 2024-03-25T18:03:41.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.175+0000 7fbf78715640 1 -- 172.21.15.18:0/2995868139 wait complete. 2024-03-25T18:03:41.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.176+0000 7fbf78715640 10 monclient: build_initial_monmap 2024-03-25T18:03:41.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.176+0000 7fbf78715640 10 monclient: monmap: 2024-03-25T18:03:41.181 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:03:41.181 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:41.182 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:03:41.182 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:03:41.182 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:41.182 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:03:41.182 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:03:41.182 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:03:41.182 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:03:41.182 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:41.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.176+0000 7fbf78715640 1 librados: starting msgr at 2024-03-25T18:03:41.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.176+0000 7fbf78715640 1 librados: starting objecter 2024-03-25T18:03:41.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.177+0000 7fbf78715640 1 Processor -- start 2024-03-25T18:03:41.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.177+0000 7fbf78715640 1 -- start start 2024-03-25T18:03:41.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.177+0000 7fbf78715640 1 librados: setting wanted keys 2024-03-25T18:03:41.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.177+0000 7fbf78715640 1 librados: calling monclient init 2024-03-25T18:03:41.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.177+0000 7fbf78715640 10 monclient: init 2024-03-25T18:03:41.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.177+0000 7fbf78715640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:41.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.177+0000 7fbf78715640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:03:41.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.177+0000 7fbf78715640 10 monclient(hunting): picked mon.noname-a con 0x7fbf700b1730 addr v1:172.21.15.18:6789/0 2024-03-25T18:03:41.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.177+0000 7fbf78715640 10 monclient(hunting): picked mon.noname-b con 0x7fbf700b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T18:03:41.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.177+0000 7fbf78715640 10 monclient(hunting): picked mon.noname-c con 0x7fbf7014be90 addr v1:172.21.15.18:6790/0 2024-03-25T18:03:41.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.177+0000 7fbf78715640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf70137760 con 0x7fbf700b1730 2024-03-25T18:03:41.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.177+0000 7fbf78715640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf701387d0 con 0x7fbf700b4d60 2024-03-25T18:03:41.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.177+0000 7fbf78715640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf70140380 con 0x7fbf7014be90 2024-03-25T18:03:41.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.177+0000 7fbf78715640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:41.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.177+0000 7fbf78715640 10 monclient(hunting): authenticate will time out at 7349.756417s 2024-03-25T18:03:41.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.197+0000 7fbf764b1640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fbf700b1730 0x7fbf70140810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:46268/0 (socket says 172.21.15.18:46268) 2024-03-25T18:03:41.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.197+0000 7fbf764b1640 1 -- 172.21.15.18:0/4105385188 learned_addr learned my addr 172.21.15.18:0/4105385188 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:03:41.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.198+0000 7fbf66ffd640 10 client.?.objecter ms_handle_connect 0x7fbf700b1730 2024-03-25T18:03:41.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.198+0000 7fbf66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:41.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.211+0000 7fbf66ffd640 10 client.?.objecter ms_handle_connect 0x7fbf700b4d60 2024-03-25T18:03:41.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.211+0000 7fbf66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:41.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.212+0000 7fbf66ffd640 10 client.?.objecter ms_handle_connect 0x7fbf7014be90 2024-03-25T18:03:41.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.212+0000 7fbf66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:41.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.219+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560047904 0 0) 0x7fbf701387d0 con 0x7fbf700b4d60 2024-03-25T18:03:41.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.219+0000 7fbf66ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:41.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.219+0000 7fbf66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:41.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.219+0000 7fbf66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:41.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.219+0000 7fbf66ffd640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:03:41.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.219+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf54004eb0 con 0x7fbf700b4d60 2024-03-25T18:03:41.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.223+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606161710 0 0) 0x7fbf70137760 con 0x7fbf700b1730 2024-03-25T18:03:41.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.223+0000 7fbf66ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:41.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.223+0000 7fbf66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:41.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.223+0000 7fbf66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:41.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.223+0000 7fbf66ffd640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:03:41.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.223+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf701387d0 con 0x7fbf700b1730 2024-03-25T18:03:41.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.228+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631431487 0 0) 0x7fbf70140380 con 0x7fbf7014be90 2024-03-25T18:03:41.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.228+0000 7fbf66ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:41.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.228+0000 7fbf66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:41.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.228+0000 7fbf66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:41.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.228+0000 7fbf66ffd640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:03:41.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.228+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf70137760 con 0x7fbf7014be90 2024-03-25T18:03:41.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.237+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 38601829 0 0) 0x7fbf54004eb0 con 0x7fbf700b4d60 2024-03-25T18:03:41.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.237+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf70140380 con 0x7fbf700b4d60 2024-03-25T18:03:41.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.237+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fbf6c0018c0 con 0x7fbf700b4d60 2024-03-25T18:03:41.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.237+0000 7fbf66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:41.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.237+0000 7fbf66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:03:41.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.237+0000 7fbf66ffd640 10 monclient(hunting): dump: 2024-03-25T18:03:41.239 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:41.239 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:41.240 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:41.240 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:41.240 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:41.240 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:41.240 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:41.240 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:41.240 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:41.240 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:41.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.237+0000 7fbf66ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:41.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.241+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4124531614 0 0) 0x7fbf70137760 con 0x7fbf7014be90 2024-03-25T18:03:41.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.241+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf54004eb0 con 0x7fbf7014be90 2024-03-25T18:03:41.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.241+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fbf68001e80 con 0x7fbf7014be90 2024-03-25T18:03:41.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.241+0000 7fbf66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:41.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.241+0000 7fbf66ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:41.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.241+0000 7fbf66ffd640 10 monclient(hunting): dump: 2024-03-25T18:03:41.244 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:41.244 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:41.244 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:41.244 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:41.244 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:41.244 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:41.244 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:41.244 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:41.244 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:41.244 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:41.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2918619840 0 0) 0x7fbf701387d0 con 0x7fbf700b1730 2024-03-25T18:03:41.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf70137760 con 0x7fbf700b1730 2024-03-25T18:03:41.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fbf600018f0 con 0x7fbf700b1730 2024-03-25T18:03:41.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:41.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf66ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:41.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf66ffd640 10 monclient(hunting): dump: 2024-03-25T18:03:41.249 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:41.249 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:41.249 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:41.249 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:41.249 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:41.249 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:41.249 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:41.249 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:41.249 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:41.249 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:41.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3057367144 0 0) 0x7fbf70140380 con 0x7fbf700b4d60 2024-03-25T18:03:41.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 >> v1:172.21.15.18:6790/0 conn(0x7fbf7014be90 legacy=0x7fbf701e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:41.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 >> v1:172.21.15.18:6789/0 conn(0x7fbf700b1730 legacy=0x7fbf70140810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:41.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf66ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:03:41.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf66ffd640 1 monclient: found mon.b 2024-03-25T18:03:41.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:41.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf66ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:03:41.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf70136910 con 0x7fbf700b4d60 2024-03-25T18:03:41.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf66ffd640 10 monclient: _finish_auth 0 2024-03-25T18:03:41.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf66ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:03:41.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:41.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf78715640 5 monclient: authenticate success, global_id 4546 2024-03-25T18:03:41.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf78715640 10 monclient: _renew_subs 2024-03-25T18:03:41.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf78715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:03:41.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf78715640 1 -- 172.21.15.18:0/4105385188 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf70136b60 con 0x7fbf700b4d60 2024-03-25T18:03:41.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf78715640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:41.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf78715640 10 monclient: _renew_subs 2024-03-25T18:03:41.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf78715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:03:41.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf78715640 1 -- 172.21.15.18:0/4105385188 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf701372b0 con 0x7fbf700b4d60 2024-03-25T18:03:41.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.245+0000 7fbf78715640 1 librados: init done 2024-03-25T18:03:41.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.246+0000 7fbf78715640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:41.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.246+0000 7fbf78715640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:41.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.246+0000 7fbf78715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:03:41.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.246+0000 7fbf78715640 1 -- 172.21.15.18:0/4105385188 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf70140510 con 0x7fbf700b4d60 2024-03-25T18:03:41.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.562+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf6c0035f0 con 0x7fbf700b4d60 2024-03-25T18:03:41.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.562+0000 7fbf66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:41.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.562+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fbf6c003900 con 0x7fbf700b4d60 2024-03-25T18:03:41.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.562+0000 7fbf66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:41.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.562+0000 7fbf66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:41.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.562+0000 7fbf66ffd640 10 monclient: dump: 2024-03-25T18:03:41.564 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:41.564 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:41.564 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:41.564 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:41.565 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:41.565 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:41.565 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:41.565 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:41.565 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:41.565 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:41.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.562+0000 7fbf74cae640 4 set_mon_vals no callback set 2024-03-25T18:03:41.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.563+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fbf6c0198d0 con 0x7fbf700b4d60 2024-03-25T18:03:41.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.563+0000 7fbf764b1640 10 monclient: get_auth_request con 0x7fbf5406e0b0 auth_method 0 2024-03-25T18:03:41.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.563+0000 7fbf66ffd640 10 client.4546.objecter ms_handle_connect 0x7fbf5406e0b0 2024-03-25T18:03:41.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.569+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fbf6c0535a0 con 0x7fbf700b4d60 2024-03-25T18:03:41.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.569+0000 7fbf66ffd640 10 client.4546.objecter ms_dispatch 0x7fbf70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:03:41.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.569+0000 7fbf66ffd640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:03:41.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.569+0000 7fbf66ffd640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:03:41.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.569+0000 7fbf66ffd640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T18:03:41.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.569+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf6c04eee0 con 0x7fbf700b4d60 2024-03-25T18:03:41.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.569+0000 7fbf66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:41.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.569+0000 7fbf66ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:41.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.833+0000 7fbf78715640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:41.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.833+0000 7fbf78715640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:41.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.833+0000 7fbf78715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:03:41.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.833+0000 7fbf78715640 1 -- 172.21.15.18:0/4105385188 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf70137550 con 0x7fbf700b4d60 2024-03-25T18:03:41.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.871+0000 7fbf66ffd640 1 -- 172.21.15.18:0/4105385188 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fbf6c050fe0 con 0x7fbf700b4d60 2024-03-25T18:03:41.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.871+0000 7fbf66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:41.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.871+0000 7fbf66ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:41.874 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:03:41.874 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:03:41.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.873+0000 7fbf78715640 10 librados: watch_flush enter 2024-03-25T18:03:41.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.873+0000 7fbf78715640 10 librados: watch_flush exit 2024-03-25T18:03:41.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.873+0000 7fbf78715640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T18:03:41.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.873+0000 7fbf78715640 10 client.4546.objecter successfully canceled tick 2024-03-25T18:03:41.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.873+0000 7fbf78715640 1 -- 172.21.15.18:0/4105385188 >> v1:172.21.15.18:6800/3721244451 conn(0x7fbf5406e0b0 legacy=0x7fbf54070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:41.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.873+0000 7fbf78715640 10 monclient: shutdown 2024-03-25T18:03:41.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.873+0000 7fbf78715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:41.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.873+0000 7fbf78715640 1 -- 172.21.15.18:0/4105385188 >> v1:172.21.15.169:6789/0 conn(0x7fbf700b4d60 legacy=0x7fbf70144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:41.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.874+0000 7fbf78715640 1 -- 172.21.15.18:0/4105385188 shutdown_connections 2024-03-25T18:03:41.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.874+0000 7fbf78715640 1 -- 172.21.15.18:0/4105385188 >> 172.21.15.18:0/4105385188 conn(0x7fbf700617a0 msgr2=0x7fbf700b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:41.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.874+0000 7fbf78715640 1 -- 172.21.15.18:0/4105385188 shutdown_connections 2024-03-25T18:03:41.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.874+0000 7fbf78715640 1 -- 172.21.15.18:0/4105385188 wait complete. 2024-03-25T18:03:41.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:41.874+0000 7fbf78715640 1 librados: shutdown 2024-03-25T18:03:41.887 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:03:48.351 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:46.143 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:48.403 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:46.195 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:48.449 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:46.197 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:48.452 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:07:46.245 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:48.458 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:46.207 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:48.471 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:46.263 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:48.476 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:07:46.224 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:48.480 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:46.272 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:48.503 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:07:46.296 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:48.693 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:03:48.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.809+0000 7fa0b740f640 10 monclient: get_monmap_and_config 2024-03-25T18:03:48.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.809+0000 7fa0b740f640 10 monclient: build_initial_monmap 2024-03-25T18:03:48.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.809+0000 7fa0b740f640 10 monclient: monmap: 2024-03-25T18:03:48.811 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:03:48.811 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:48.811 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:03:48.810889+0000 2024-03-25T18:03:48.811 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:03:48.810889+0000 2024-03-25T18:03:48.811 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:48.811 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:48.812 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:03:48.812 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:03:48.812 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:03:48.812 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:48.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.810+0000 7fa0b740f640 1 Processor -- start 2024-03-25T18:03:48.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.810+0000 7fa0b740f640 1 -- start start 2024-03-25T18:03:48.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.810+0000 7fa0b740f640 10 monclient: init 2024-03-25T18:03:48.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.811+0000 7fa0b740f640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:48.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.811+0000 7fa0b740f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:03:48.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.811+0000 7fa0b740f640 10 monclient(hunting): picked mon.noname-a con 0x7fa0b00a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:03:48.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.811+0000 7fa0b740f640 10 monclient(hunting): picked mon.noname-c con 0x7fa0b013f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:03:48.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.811+0000 7fa0b740f640 10 monclient(hunting): picked mon.noname-b con 0x7fa0b01430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:03:48.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.811+0000 7fa0b740f640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b0142520 con 0x7fa0b00a98f0 2024-03-25T18:03:48.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.811+0000 7fa0b740f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b00b28a0 con 0x7fa0b013f710 2024-03-25T18:03:48.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.811+0000 7fa0b740f640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b00b2a30 con 0x7fa0b01430c0 2024-03-25T18:03:48.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.811+0000 7fa0b740f640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:48.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.811+0000 7fa0b740f640 10 monclient(hunting): authenticate will time out at 7357.390016s 2024-03-25T18:03:48.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.816+0000 7fa0b51ab640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fa0b00a98f0 0x7fa0b00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:46310/0 (socket says 172.21.15.18:46310) 2024-03-25T18:03:48.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.816+0000 7fa0b51ab640 1 -- 172.21.15.18:0/2365221024 learned_addr learned my addr 172.21.15.18:0/2365221024 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:03:48.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.829+0000 7fa0a7fff640 1 -- 172.21.15.18:0/2365221024 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479287406 0 0) 0x7fa0b0142520 con 0x7fa0b00a98f0 2024-03-25T18:03:48.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.829+0000 7fa0a7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:03:48.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.829+0000 7fa0a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:48.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.829+0000 7fa0a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:48.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.829+0000 7fa0a7fff640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:03:48.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.829+0000 7fa0a7fff640 1 -- 172.21.15.18:0/2365221024 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa09c003520 con 0x7fa0b00a98f0 2024-03-25T18:03:48.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.833+0000 7fa0a7fff640 1 -- 172.21.15.18:0/2365221024 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695061963 0 0) 0x7fa0b00b28a0 con 0x7fa0b013f710 2024-03-25T18:03:48.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.833+0000 7fa0a7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:03:48.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.833+0000 7fa0a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:48.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.833+0000 7fa0a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:48.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.833+0000 7fa0a7fff640 10 monclient(hunting): my global_id is 4549 2024-03-25T18:03:48.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.833+0000 7fa0a7fff640 1 -- 172.21.15.18:0/2365221024 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0b0142520 con 0x7fa0b013f710 2024-03-25T18:03:48.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.847+0000 7fa0a7fff640 1 -- 172.21.15.18:0/2365221024 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429518281 0 0) 0x7fa0b00b2a30 con 0x7fa0b01430c0 2024-03-25T18:03:48.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.847+0000 7fa0a7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:03:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.847+0000 7fa0a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.847+0000 7fa0a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.847+0000 7fa0a7fff640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:03:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.847+0000 7fa0a7fff640 1 -- 172.21.15.18:0/2365221024 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0b00b28a0 con 0x7fa0b01430c0 2024-03-25T18:03:48.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.857+0000 7fa0a7fff640 1 -- 172.21.15.18:0/2365221024 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3801105736 0 0) 0x7fa09c003520 con 0x7fa0b00a98f0 2024-03-25T18:03:48.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.857+0000 7fa0a7fff640 1 -- 172.21.15.18:0/2365221024 >> v1:172.21.15.18:6790/0 conn(0x7fa0b01430c0 legacy=0x7fa0b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:48.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.857+0000 7fa0a7fff640 1 -- 172.21.15.18:0/2365221024 >> v1:172.21.15.169:6789/0 conn(0x7fa0b013f710 legacy=0x7fa0b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:48.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.857+0000 7fa0a7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:03:48.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.857+0000 7fa0a7fff640 1 monclient: found mon.noname-a 2024-03-25T18:03:48.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.857+0000 7fa0a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:48.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.857+0000 7fa0a7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:03:48.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.857+0000 7fa0a7fff640 1 -- 172.21.15.18:0/2365221024 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b00b44c0 con 0x7fa0b00a98f0 2024-03-25T18:03:48.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.857+0000 7fa0a7fff640 10 monclient: _finish_auth 0 2024-03-25T18:03:48.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.857+0000 7fa0a7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:03:48.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.857+0000 7fa0a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:48.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.857+0000 7fa0a7fff640 1 -- 172.21.15.18:0/2365221024 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa0ac001690 con 0x7fa0b00a98f0 2024-03-25T18:03:48.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.857+0000 7fa0a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:48.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.857+0000 7fa0a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:03:48.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.857+0000 7fa0a7fff640 10 monclient: dump: 2024-03-25T18:03:48.861 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:48.861 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:48.861 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:48.862 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:48.862 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:48.862 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:48.862 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:48.862 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:48.862 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:48.862 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:48.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.857+0000 7fa0b740f640 5 monclient: authenticate success, global_id 4563 2024-03-25T18:03:48.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.857+0000 7fa0b740f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:48.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.870+0000 7fa0a7fff640 1 -- 172.21.15.18:0/2365221024 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0ac0017f0 con 0x7fa0b00a98f0 2024-03-25T18:03:48.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.870+0000 7fa0a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:48.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.870+0000 7fa0a7fff640 1 -- 172.21.15.18:0/2365221024 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa0ac001c00 con 0x7fa0b00a98f0 2024-03-25T18:03:48.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.870+0000 7fa0b740f640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:48.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.870+0000 7fa0b740f640 4 set_mon_vals no callback set 2024-03-25T18:03:48.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.870+0000 7fa0a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:48.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.870+0000 7fa0a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:48.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.870+0000 7fa0a7fff640 10 monclient: dump: 2024-03-25T18:03:48.873 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:48.873 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:48.873 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:48.873 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:48.873 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:48.873 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:48.873 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:48.873 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:48.874 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:48.874 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:48.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.871+0000 7fa0b740f640 10 monclient: shutdown 2024-03-25T18:03:48.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.871+0000 7fa0b740f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:48.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.871+0000 7fa0b740f640 1 -- 172.21.15.18:0/2365221024 >> v1:172.21.15.18:6789/0 conn(0x7fa0b00a98f0 legacy=0x7fa0b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:48.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.871+0000 7fa0b740f640 1 -- 172.21.15.18:0/2365221024 shutdown_connections 2024-03-25T18:03:48.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.871+0000 7fa0b740f640 1 -- 172.21.15.18:0/2365221024 >> 172.21.15.18:0/2365221024 conn(0x7fa0b00b7ea0 msgr2=0x7fa0b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:48.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.871+0000 7fa0b740f640 1 -- 172.21.15.18:0/2365221024 shutdown_connections 2024-03-25T18:03:48.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.871+0000 7fa0b740f640 1 -- 172.21.15.18:0/2365221024 wait complete. 2024-03-25T18:03:48.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.872+0000 7fa0b740f640 10 monclient: build_initial_monmap 2024-03-25T18:03:48.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.872+0000 7fa0b740f640 10 monclient: monmap: 2024-03-25T18:03:48.877 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:03:48.877 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:48.877 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:03:48.877 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:03:48.877 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:48.877 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:03:48.877 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:03:48.877 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:03:48.877 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:03:48.877 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:48.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.872+0000 7fa0b740f640 1 librados: starting msgr at 2024-03-25T18:03:48.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.872+0000 7fa0b740f640 1 librados: starting objecter 2024-03-25T18:03:48.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.872+0000 7fa0b740f640 1 Processor -- start 2024-03-25T18:03:48.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.872+0000 7fa0b740f640 1 -- start start 2024-03-25T18:03:48.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.872+0000 7fa0b740f640 1 librados: setting wanted keys 2024-03-25T18:03:48.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.872+0000 7fa0b740f640 1 librados: calling monclient init 2024-03-25T18:03:48.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.872+0000 7fa0b740f640 10 monclient: init 2024-03-25T18:03:48.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.873+0000 7fa0b740f640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:48.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.873+0000 7fa0b740f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:03:48.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.873+0000 7fa0b740f640 10 monclient(hunting): picked mon.noname-c con 0x7fa0b00a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:03:48.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.873+0000 7fa0b740f640 10 monclient(hunting): picked mon.noname-b con 0x7fa0b013f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:03:48.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.873+0000 7fa0b740f640 10 monclient(hunting): picked mon.noname-a con 0x7fa0b01430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:03:48.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.873+0000 7fa0b740f640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b00abd30 con 0x7fa0b01430c0 2024-03-25T18:03:48.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.873+0000 7fa0b740f640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b01d77a0 con 0x7fa0b013f710 2024-03-25T18:03:48.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.873+0000 7fa0b740f640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b01d51f0 con 0x7fa0b00a98f0 2024-03-25T18:03:48.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.873+0000 7fa0b740f640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:48.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.873+0000 7fa0b740f640 10 monclient(hunting): authenticate will time out at 7357.452642s 2024-03-25T18:03:48.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.883+0000 7fa0b49aa640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fa0b013f710 0x7fa0b0138340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:60670/0 (socket says 172.21.15.18:60670) 2024-03-25T18:03:48.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.883+0000 7fa0b49aa640 1 -- 172.21.15.18:0/742885895 learned_addr learned my addr 172.21.15.18:0/742885895 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:03:48.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.889+0000 7fa0a5ffb640 10 client.?.objecter ms_handle_connect 0x7fa0b013f710 2024-03-25T18:03:48.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.889+0000 7fa0a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:48.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.889+0000 7fa0a5ffb640 10 client.?.objecter ms_handle_connect 0x7fa0b00a98f0 2024-03-25T18:03:48.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.889+0000 7fa0a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:48.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.894+0000 7fa0a5ffb640 1 -- 172.21.15.18:0/742885895 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783613528 0 0) 0x7fa0b01d51f0 con 0x7fa0b00a98f0 2024-03-25T18:03:48.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.894+0000 7fa0a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:03:48.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.894+0000 7fa0a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:48.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.894+0000 7fa0a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:48.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.894+0000 7fa0a5ffb640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:03:48.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.894+0000 7fa0a5ffb640 1 -- 172.21.15.18:0/742885895 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa094004eb0 con 0x7fa0b00a98f0 2024-03-25T18:03:48.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.902+0000 7fa0a5ffb640 10 client.?.objecter ms_handle_connect 0x7fa0b01430c0 2024-03-25T18:03:48.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.902+0000 7fa0a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:48.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.903+0000 7fa0a5ffb640 1 -- 172.21.15.18:0/742885895 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2761494751 0 0) 0x7fa094004eb0 con 0x7fa0b00a98f0 2024-03-25T18:03:48.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.903+0000 7fa0a5ffb640 1 -- 172.21.15.18:0/742885895 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0b01d51f0 con 0x7fa0b00a98f0 2024-03-25T18:03:48.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.903+0000 7fa0a5ffb640 1 -- 172.21.15.18:0/742885895 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa0ac001b00 con 0x7fa0b00a98f0 2024-03-25T18:03:48.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.903+0000 7fa0a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:48.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.903+0000 7fa0a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:48.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.903+0000 7fa0a5ffb640 10 monclient(hunting): dump: 2024-03-25T18:03:48.905 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:48.905 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:48.905 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:48.905 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:48.905 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:48.905 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:48.905 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:48.906 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:48.906 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:48.906 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:48.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.903+0000 7fa0a5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:48.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.907+0000 7fa0a5ffb640 1 -- 172.21.15.18:0/742885895 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308032931 0 0) 0x7fa0b01d77a0 con 0x7fa0b013f710 2024-03-25T18:03:48.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.908+0000 7fa0a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:03:48.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.908+0000 7fa0a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:48.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.908+0000 7fa0a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:48.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.908+0000 7fa0a5ffb640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:03:48.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.908+0000 7fa0a5ffb640 1 -- 172.21.15.18:0/742885895 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa094004eb0 con 0x7fa0b013f710 2024-03-25T18:03:48.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0a5ffb640 1 -- 172.21.15.18:0/742885895 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3890106299 0 0) 0x7fa0b01d51f0 con 0x7fa0b00a98f0 2024-03-25T18:03:48.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0a5ffb640 1 -- 172.21.15.18:0/742885895 >> v1:172.21.15.169:6789/0 conn(0x7fa0b013f710 legacy=0x7fa0b0138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:48.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0a5ffb640 1 -- 172.21.15.18:0/742885895 >> v1:172.21.15.18:6789/0 conn(0x7fa0b01430c0 legacy=0x7fa0b01e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:48.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0a5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:03:48.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0a5ffb640 1 monclient: found mon.c 2024-03-25T18:03:48.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:48.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0a5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:48.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0a5ffb640 1 -- 172.21.15.18:0/742885895 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b01d83d0 con 0x7fa0b00a98f0 2024-03-25T18:03:48.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0a5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:03:48.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0a5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:03:48.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:48.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0b740f640 5 monclient: authenticate success, global_id 4571 2024-03-25T18:03:48.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0b740f640 10 monclient: _renew_subs 2024-03-25T18:03:48.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0b740f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:48.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0b740f640 1 -- 172.21.15.18:0/742885895 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b0138a50 con 0x7fa0b00a98f0 2024-03-25T18:03:48.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0b740f640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:48.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0b740f640 10 monclient: _renew_subs 2024-03-25T18:03:48.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0b740f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:48.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0b740f640 1 -- 172.21.15.18:0/742885895 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b01e35c0 con 0x7fa0b00a98f0 2024-03-25T18:03:48.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.918+0000 7fa0b740f640 1 librados: init done 2024-03-25T18:03:48.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.920+0000 7fa0b740f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:48.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.920+0000 7fa0b740f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:48.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.920+0000 7fa0b740f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:48.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:48.920+0000 7fa0b740f640 1 -- 172.21.15.18:0/742885895 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b00627b0 con 0x7fa0b00a98f0 2024-03-25T18:03:49.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.223+0000 7fa0a5ffb640 1 -- 172.21.15.18:0/742885895 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0ac001220 con 0x7fa0b00a98f0 2024-03-25T18:03:49.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.223+0000 7fa0a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:49.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.223+0000 7fa0a5ffb640 1 -- 172.21.15.18:0/742885895 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa0ac004c30 con 0x7fa0b00a98f0 2024-03-25T18:03:49.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.223+0000 7fa0a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:49.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.223+0000 7fa0a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:49.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.223+0000 7fa0a5ffb640 10 monclient: dump: 2024-03-25T18:03:49.225 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:49.226 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:49.226 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:49.226 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:49.226 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:49.226 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:49.226 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:49.226 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:49.226 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:49.226 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:49.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.223+0000 7fa0a77fe640 4 set_mon_vals no callback set 2024-03-25T18:03:49.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.224+0000 7fa0a5ffb640 1 -- 172.21.15.18:0/742885895 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fa0ac01ac40 con 0x7fa0b00a98f0 2024-03-25T18:03:49.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.224+0000 7fa0b49aa640 10 monclient: get_auth_request con 0x7fa09406dd60 auth_method 0 2024-03-25T18:03:49.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.224+0000 7fa0a5ffb640 10 client.4571.objecter ms_handle_connect 0x7fa09406dd60 2024-03-25T18:03:49.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.233+0000 7fa0a5ffb640 1 -- 172.21.15.18:0/742885895 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fa0ac0548b0 con 0x7fa0b00a98f0 2024-03-25T18:03:49.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.233+0000 7fa0a5ffb640 10 client.4571.objecter ms_dispatch 0x7fa0b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:03:49.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.233+0000 7fa0a5ffb640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:03:49.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.233+0000 7fa0a5ffb640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:03:49.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.233+0000 7fa0a5ffb640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-25T18:03:49.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.233+0000 7fa0a5ffb640 1 -- 172.21.15.18:0/742885895 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0ac0501f0 con 0x7fa0b00a98f0 2024-03-25T18:03:49.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.233+0000 7fa0a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:49.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.233+0000 7fa0a5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:49.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.458+0000 7fa0b740f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:49.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.458+0000 7fa0b740f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:49.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.458+0000 7fa0b740f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:49.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.458+0000 7fa0b740f640 1 -- 172.21.15.18:0/742885895 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0b013e250 con 0x7fa0b00a98f0 2024-03-25T18:03:49.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.573+0000 7fa0a5ffb640 1 -- 172.21.15.18:0/742885895 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fa0ac0522f0 con 0x7fa0b00a98f0 2024-03-25T18:03:49.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.573+0000 7fa0a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:49.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.573+0000 7fa0a5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:49.575 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:03:49.575 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:03:49.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.575+0000 7fa0b740f640 10 librados: watch_flush enter 2024-03-25T18:03:49.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.575+0000 7fa0b740f640 10 librados: watch_flush exit 2024-03-25T18:03:49.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.575+0000 7fa0b740f640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-25T18:03:49.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.576+0000 7fa0b740f640 10 client.4571.objecter successfully canceled tick 2024-03-25T18:03:49.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.576+0000 7fa0b740f640 1 -- 172.21.15.18:0/742885895 >> v1:172.21.15.18:6800/3721244451 conn(0x7fa09406dd60 legacy=0x7fa094070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:49.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.576+0000 7fa0b740f640 10 monclient: shutdown 2024-03-25T18:03:49.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.576+0000 7fa0b740f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:49.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.576+0000 7fa0b740f640 1 -- 172.21.15.18:0/742885895 >> v1:172.21.15.18:6790/0 conn(0x7fa0b00a98f0 legacy=0x7fa0b01d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:49.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.576+0000 7fa0b740f640 1 -- 172.21.15.18:0/742885895 shutdown_connections 2024-03-25T18:03:49.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.576+0000 7fa0b740f640 1 -- 172.21.15.18:0/742885895 >> 172.21.15.18:0/742885895 conn(0x7fa0b00b7ea0 msgr2=0x7fa0b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:49.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.576+0000 7fa0b740f640 1 -- 172.21.15.18:0/742885895 shutdown_connections 2024-03-25T18:03:49.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.576+0000 7fa0b740f640 1 -- 172.21.15.18:0/742885895 wait complete. 2024-03-25T18:03:49.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:49.576+0000 7fa0b740f640 1 librados: shutdown 2024-03-25T18:03:49.588 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:03:56.395 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:03:56.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.504+0000 7f85f8255640 10 monclient: get_monmap_and_config 2024-03-25T18:03:56.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.504+0000 7f85f8255640 10 monclient: build_initial_monmap 2024-03-25T18:03:56.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.504+0000 7f85f8255640 10 monclient: monmap: 2024-03-25T18:03:56.506 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:03:56.506 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:56.506 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:03:56.505850+0000 2024-03-25T18:03:56.506 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:03:56.505850+0000 2024-03-25T18:03:56.506 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:56.506 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:56.506 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:03:56.506 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:03:56.506 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:03:56.506 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:56.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.505+0000 7f85f8255640 1 Processor -- start 2024-03-25T18:03:56.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.505+0000 7f85f8255640 1 -- start start 2024-03-25T18:03:56.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.505+0000 7f85f8255640 10 monclient: init 2024-03-25T18:03:56.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.506+0000 7f85f8255640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:56.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.506+0000 7f85f8255640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:03:56.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.506+0000 7f85f8255640 10 monclient(hunting): picked mon.noname-c con 0x7f85f00b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T18:03:56.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.506+0000 7f85f8255640 10 monclient(hunting): picked mon.noname-b con 0x7f85f00b1730 addr v1:172.21.15.18:6790/0 2024-03-25T18:03:56.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.506+0000 7f85f8255640 10 monclient(hunting): picked mon.noname-a con 0x7f85f014be90 addr v1:172.21.15.18:6789/0 2024-03-25T18:03:56.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.506+0000 7f85f8255640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f00af940 con 0x7f85f014be90 2024-03-25T18:03:56.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.506+0000 7f85f8255640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f00b0a70 con 0x7f85f00b4d60 2024-03-25T18:03:56.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.506+0000 7f85f8255640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f00b0d50 con 0x7f85f00b1730 2024-03-25T18:03:56.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.506+0000 7f85f8255640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:56.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.506+0000 7f85f8255640 10 monclient(hunting): authenticate will time out at 7365.085130s 2024-03-25T18:03:56.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.514+0000 7f85f67f2640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f85f014be90 0x7f85f014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:44240/0 (socket says 172.21.15.18:44240) 2024-03-25T18:03:56.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.514+0000 7f85f67f2640 1 -- 172.21.15.18:0/2563551142 learned_addr learned my addr 172.21.15.18:0/2563551142 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:03:56.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.526+0000 7f85f4fef640 1 -- 172.21.15.18:0/2563551142 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137939780 0 0) 0x7f85f00b0d50 con 0x7f85f00b1730 2024-03-25T18:03:56.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.526+0000 7f85f4fef640 10 monclient(hunting): _negotiate 2024-03-25T18:03:56.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.526+0000 7f85f4fef640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:56.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.526+0000 7f85f4fef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:56.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.526+0000 7f85f4fef640 10 monclient(hunting): my global_id is 4574 2024-03-25T18:03:56.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.526+0000 7f85f4fef640 1 -- 172.21.15.18:0/2563551142 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85dc003520 con 0x7f85f00b1730 2024-03-25T18:03:56.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.531+0000 7f85f4fef640 1 -- 172.21.15.18:0/2563551142 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270748185 0 0) 0x7f85f00af940 con 0x7f85f014be90 2024-03-25T18:03:56.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.531+0000 7f85f4fef640 10 monclient(hunting): _negotiate 2024-03-25T18:03:56.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.531+0000 7f85f4fef640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:56.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.531+0000 7f85f4fef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:56.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.531+0000 7f85f4fef640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:03:56.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.532+0000 7f85f4fef640 1 -- 172.21.15.18:0/2563551142 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f00b0d50 con 0x7f85f014be90 2024-03-25T18:03:56.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.541+0000 7f85f4fef640 1 -- 172.21.15.18:0/2563551142 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3115525486 0 0) 0x7f85dc003520 con 0x7f85f00b1730 2024-03-25T18:03:56.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.541+0000 7f85f4fef640 1 -- 172.21.15.18:0/2563551142 >> v1:172.21.15.169:6789/0 conn(0x7f85f00b4d60 legacy=0x7f85f013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:56.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.541+0000 7f85f4fef640 1 -- 172.21.15.18:0/2563551142 >> v1:172.21.15.18:6789/0 conn(0x7f85f014be90 legacy=0x7f85f014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:56.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.541+0000 7f85f4fef640 10 monclient: _finish_hunting 0 2024-03-25T18:03:56.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.541+0000 7f85f4fef640 1 monclient: found mon.noname-b 2024-03-25T18:03:56.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.541+0000 7f85f4fef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:56.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.541+0000 7f85f4fef640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:03:56.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.541+0000 7f85f4fef640 1 -- 172.21.15.18:0/2563551142 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f00b1030 con 0x7f85f00b1730 2024-03-25T18:03:56.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.541+0000 7f85f4fef640 10 monclient: _finish_auth 0 2024-03-25T18:03:56.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.541+0000 7f85f4fef640 10 monclient: _check_auth_tickets 2024-03-25T18:03:56.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.541+0000 7f85f4fef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:56.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.541+0000 7f85f4fef640 1 -- 172.21.15.18:0/2563551142 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f85e0001690 con 0x7f85f00b1730 2024-03-25T18:03:56.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.541+0000 7f85f4fef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:56.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.541+0000 7f85f4fef640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:03:56.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.541+0000 7f85f4fef640 10 monclient: dump: 2024-03-25T18:03:56.544 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:56.544 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:56.544 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:56.544 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:56.544 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:56.544 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:56.544 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:56.544 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:56.544 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:56.545 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:56.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.541+0000 7f85f8255640 5 monclient: authenticate success, global_id 4574 2024-03-25T18:03:56.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.541+0000 7f85f8255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:56.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.557+0000 7f85f4fef640 1 -- 172.21.15.18:0/2563551142 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85e00017f0 con 0x7f85f00b1730 2024-03-25T18:03:56.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.557+0000 7f85f4fef640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:56.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.557+0000 7f85f4fef640 1 -- 172.21.15.18:0/2563551142 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f85e0001c00 con 0x7f85f00b1730 2024-03-25T18:03:56.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.557+0000 7f85f4fef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:56.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.557+0000 7f85f4fef640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:56.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.557+0000 7f85f4fef640 10 monclient: dump: 2024-03-25T18:03:56.559 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:56.559 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:56.559 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:56.559 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:56.559 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:56.559 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:56.559 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:56.559 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:56.559 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:56.559 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:56.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.557+0000 7f85f8255640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:56.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.557+0000 7f85f8255640 4 set_mon_vals no callback set 2024-03-25T18:03:56.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.558+0000 7f85f8255640 10 monclient: shutdown 2024-03-25T18:03:56.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.558+0000 7f85f8255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:56.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.558+0000 7f85f8255640 1 -- 172.21.15.18:0/2563551142 >> v1:172.21.15.18:6790/0 conn(0x7f85f00b1730 legacy=0x7f85f00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:56.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.558+0000 7f85f8255640 1 -- 172.21.15.18:0/2563551142 shutdown_connections 2024-03-25T18:03:56.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.558+0000 7f85f8255640 1 -- 172.21.15.18:0/2563551142 >> 172.21.15.18:0/2563551142 conn(0x7f85f00617a0 msgr2=0x7f85f0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:56.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.558+0000 7f85f8255640 1 -- 172.21.15.18:0/2563551142 shutdown_connections 2024-03-25T18:03:56.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.558+0000 7f85f8255640 1 -- 172.21.15.18:0/2563551142 wait complete. 2024-03-25T18:03:56.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.558+0000 7f85f8255640 10 monclient: build_initial_monmap 2024-03-25T18:03:56.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.558+0000 7f85f8255640 10 monclient: monmap: 2024-03-25T18:03:56.562 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:03:56.562 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:56.562 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:03:56.562 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:03:56.562 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:56.562 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:03:56.562 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:03:56.562 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:03:56.563 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:03:56.563 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:56.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.559+0000 7f85f8255640 1 librados: starting msgr at 2024-03-25T18:03:56.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.559+0000 7f85f8255640 1 librados: starting objecter 2024-03-25T18:03:56.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.559+0000 7f85f8255640 1 Processor -- start 2024-03-25T18:03:56.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.559+0000 7f85f8255640 1 -- start start 2024-03-25T18:03:56.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.559+0000 7f85f8255640 1 librados: setting wanted keys 2024-03-25T18:03:56.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.559+0000 7f85f8255640 1 librados: calling monclient init 2024-03-25T18:03:56.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.559+0000 7f85f8255640 10 monclient: init 2024-03-25T18:03:56.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.560+0000 7f85f8255640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:56.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.560+0000 7f85f8255640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:03:56.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.560+0000 7f85f8255640 10 monclient(hunting): picked mon.noname-c con 0x7f85f00b1730 addr v1:172.21.15.18:6790/0 2024-03-25T18:03:56.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.560+0000 7f85f8255640 10 monclient(hunting): picked mon.noname-a con 0x7f85f00b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T18:03:56.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.560+0000 7f85f8255640 10 monclient(hunting): picked mon.noname-b con 0x7f85f014be90 addr v1:172.21.15.169:6789/0 2024-03-25T18:03:56.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.560+0000 7f85f8255640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f0140740 con 0x7f85f00b4d60 2024-03-25T18:03:56.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.560+0000 7f85f8255640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f0140380 con 0x7f85f014be90 2024-03-25T18:03:56.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.560+0000 7f85f8255640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f0140510 con 0x7f85f00b1730 2024-03-25T18:03:56.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.560+0000 7f85f8255640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:56.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.560+0000 7f85f8255640 10 monclient(hunting): authenticate will time out at 7365.138901s 2024-03-25T18:03:56.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.569+0000 7f85f67f2640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f85f014be90 0x7f85f01db3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:47472/0 (socket says 172.21.15.18:47472) 2024-03-25T18:03:56.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.569+0000 7f85f67f2640 1 -- 172.21.15.18:0/650036499 learned_addr learned my addr 172.21.15.18:0/650036499 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:03:56.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.570+0000 7f85e67fc640 10 client.?.objecter ms_handle_connect 0x7f85f014be90 2024-03-25T18:03:56.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.570+0000 7f85e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:56.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.574+0000 7f85e67fc640 10 client.?.objecter ms_handle_connect 0x7f85f00b1730 2024-03-25T18:03:56.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.574+0000 7f85e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:56.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.582+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824855176 0 0) 0x7f85f0140510 con 0x7f85f00b1730 2024-03-25T18:03:56.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.582+0000 7f85e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:03:56.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.582+0000 7f85e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:56.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.582+0000 7f85e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:56.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.582+0000 7f85e67fc640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:03:56.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.582+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85d4004eb0 con 0x7f85f00b1730 2024-03-25T18:03:56.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.586+0000 7f85e67fc640 10 client.?.objecter ms_handle_connect 0x7f85f00b4d60 2024-03-25T18:03:56.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.586+0000 7f85e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:56.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.590+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749975293 0 0) 0x7f85f0140380 con 0x7f85f014be90 2024-03-25T18:03:56.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.590+0000 7f85e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:03:56.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.590+0000 7f85e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:56.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.590+0000 7f85e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:56.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.590+0000 7f85e67fc640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:03:56.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.590+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f0140510 con 0x7f85f014be90 2024-03-25T18:03:56.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.600+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776666982 0 0) 0x7f85f0140740 con 0x7f85f00b4d60 2024-03-25T18:03:56.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.601+0000 7f85e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:03:56.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.601+0000 7f85e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:56.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.601+0000 7f85e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:56.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.601+0000 7f85e67fc640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:03:56.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.601+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f0140380 con 0x7f85f00b4d60 2024-03-25T18:03:56.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.602+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1270190465 0 0) 0x7f85d4004eb0 con 0x7f85f00b1730 2024-03-25T18:03:56.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.602+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85f0140740 con 0x7f85f00b1730 2024-03-25T18:03:56.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.602+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f85ec0018c0 con 0x7f85f00b1730 2024-03-25T18:03:56.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.602+0000 7f85e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:56.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.602+0000 7f85e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:56.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.602+0000 7f85e67fc640 10 monclient(hunting): dump: 2024-03-25T18:03:56.604 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:56.604 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:56.604 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:56.604 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:56.604 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:56.604 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:56.605 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:56.605 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:56.605 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:56.605 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:56.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.602+0000 7f85e67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:56.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.606+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2478476516 0 0) 0x7f85f0140510 con 0x7f85f014be90 2024-03-25T18:03:56.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.606+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85d4004eb0 con 0x7f85f014be90 2024-03-25T18:03:56.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.606+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f85e8001c10 con 0x7f85f014be90 2024-03-25T18:03:56.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.606+0000 7f85e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:56.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.606+0000 7f85e67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:56.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.606+0000 7f85e67fc640 10 monclient(hunting): dump: 2024-03-25T18:03:56.608 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:56.608 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:56.608 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:56.608 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:56.609 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:56.609 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:56.609 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:56.609 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:56.609 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:56.609 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:56.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 28273950 0 0) 0x7f85f0140740 con 0x7f85f00b1730 2024-03-25T18:03:56.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 >> v1:172.21.15.169:6789/0 conn(0x7f85f014be90 legacy=0x7f85f01db3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:56.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 >> v1:172.21.15.18:6789/0 conn(0x7f85f00b4d60 legacy=0x7f85f01daca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:56.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85e67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:03:56.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85e67fc640 1 monclient: found mon.c 2024-03-25T18:03:56.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:56.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85e67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:56.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f0137400 con 0x7f85f00b1730 2024-03-25T18:03:56.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85e67fc640 10 monclient: _finish_auth 0 2024-03-25T18:03:56.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85e67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:03:56.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:56.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85f8255640 5 monclient: authenticate success, global_id 4577 2024-03-25T18:03:56.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85f8255640 10 monclient: _renew_subs 2024-03-25T18:03:56.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85f8255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:56.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85f8255640 1 -- 172.21.15.18:0/650036499 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f0137610 con 0x7f85f00b1730 2024-03-25T18:03:56.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85f8255640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:56.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85f8255640 10 monclient: _renew_subs 2024-03-25T18:03:56.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85f8255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:56.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85f8255640 1 -- 172.21.15.18:0/650036499 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f0137f80 con 0x7f85f00b1730 2024-03-25T18:03:56.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.613+0000 7f85f8255640 1 librados: init done 2024-03-25T18:03:56.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.614+0000 7f85f8255640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:56.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.614+0000 7f85f8255640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:56.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.614+0000 7f85f8255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:56.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.614+0000 7f85f8255640 1 -- 172.21.15.18:0/650036499 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f01401f0 con 0x7f85f00b1730 2024-03-25T18:03:56.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.934+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85ec0035f0 con 0x7f85f00b1730 2024-03-25T18:03:56.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.934+0000 7f85e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:56.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.934+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f85ec003900 con 0x7f85f00b1730 2024-03-25T18:03:56.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.934+0000 7f85e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:56.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.934+0000 7f85e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:56.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.934+0000 7f85e67fc640 10 monclient: dump: 2024-03-25T18:03:56.936 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:03:56.937 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:03:56.937 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:56.937 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:03:56.937 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:56.937 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:03:56.937 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:03:56.937 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:03:56.937 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:03:56.937 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:03:56.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.934+0000 7f85e7fff640 4 set_mon_vals no callback set 2024-03-25T18:03:56.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.935+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f85ec0198d0 con 0x7f85f00b1730 2024-03-25T18:03:56.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.936+0000 7f85f57f0640 10 monclient: get_auth_request con 0x7f85d406df20 auth_method 0 2024-03-25T18:03:56.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.936+0000 7f85e67fc640 10 client.4577.objecter ms_handle_connect 0x7f85d406df20 2024-03-25T18:03:56.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.942+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f85ec0535b0 con 0x7f85f00b1730 2024-03-25T18:03:56.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.942+0000 7f85e67fc640 10 client.4577.objecter ms_dispatch 0x7f85f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:03:56.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.942+0000 7f85e67fc640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:03:56.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.942+0000 7f85e67fc640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:03:56.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.942+0000 7f85e67fc640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-25T18:03:56.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.942+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f85ec04eef0 con 0x7f85f00b1730 2024-03-25T18:03:56.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.942+0000 7f85e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:56.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:56.942+0000 7f85e67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:57.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.166+0000 7f85f8255640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:57.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.166+0000 7f85f8255640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:57.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.166+0000 7f85f8255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:03:57.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.166+0000 7f85f8255640 1 -- 172.21.15.18:0/650036499 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85f01408d0 con 0x7f85f00b1730 2024-03-25T18:03:57.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.225+0000 7f85e67fc640 1 -- 172.21.15.18:0/650036499 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f85ec050ff0 con 0x7f85f00b1730 2024-03-25T18:03:57.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.225+0000 7f85e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:57.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.225+0000 7f85e67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:57.227 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:03:57.227 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:03:57.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.227+0000 7f85f8255640 10 librados: watch_flush enter 2024-03-25T18:03:57.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.227+0000 7f85f8255640 10 librados: watch_flush exit 2024-03-25T18:03:57.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.228+0000 7f85f8255640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-25T18:03:57.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.228+0000 7f85f8255640 10 client.4577.objecter successfully canceled tick 2024-03-25T18:03:57.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.228+0000 7f85f8255640 1 -- 172.21.15.18:0/650036499 >> v1:172.21.15.18:6800/3721244451 conn(0x7f85d406df20 legacy=0x7f85d40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:57.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.228+0000 7f85f8255640 10 monclient: shutdown 2024-03-25T18:03:57.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.228+0000 7f85f8255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:57.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.228+0000 7f85f8255640 1 -- 172.21.15.18:0/650036499 >> v1:172.21.15.18:6790/0 conn(0x7f85f00b1730 legacy=0x7f85f0134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:57.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.228+0000 7f85f8255640 1 -- 172.21.15.18:0/650036499 shutdown_connections 2024-03-25T18:03:57.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.228+0000 7f85f8255640 1 -- 172.21.15.18:0/650036499 >> 172.21.15.18:0/650036499 conn(0x7f85f00617a0 msgr2=0x7f85f00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:57.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.228+0000 7f85f8255640 1 -- 172.21.15.18:0/650036499 shutdown_connections 2024-03-25T18:03:57.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.228+0000 7f85f8255640 1 -- 172.21.15.18:0/650036499 wait complete. 2024-03-25T18:03:57.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:03:57.228+0000 7f85f8255640 1 librados: shutdown 2024-03-25T18:03:57.241 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:04:04.048 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:04:04.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.165+0000 7f45db625640 10 monclient: get_monmap_and_config 2024-03-25T18:04:04.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.165+0000 7f45db625640 10 monclient: build_initial_monmap 2024-03-25T18:04:04.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.165+0000 7f45db625640 10 monclient: monmap: 2024-03-25T18:04:04.166 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:04:04.167 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:04.167 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:04:04.166512+0000 2024-03-25T18:04:04.167 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:04:04.166512+0000 2024-03-25T18:04:04.167 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:04.167 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:04.167 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:04:04.167 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:04:04.167 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:04:04.167 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:04.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.165+0000 7f45db625640 1 Processor -- start 2024-03-25T18:04:04.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.166+0000 7f45db625640 1 -- start start 2024-03-25T18:04:04.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.166+0000 7f45db625640 10 monclient: init 2024-03-25T18:04:04.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.166+0000 7f45db625640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:04.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.166+0000 7f45db625640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:04:04.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.166+0000 7f45db625640 10 monclient(hunting): picked mon.noname-a con 0x7f45d40a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:04:04.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.166+0000 7f45db625640 10 monclient(hunting): picked mon.noname-c con 0x7f45d413f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:04:04.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.166+0000 7f45db625640 10 monclient(hunting): picked mon.noname-b con 0x7f45d41430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:04:04.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.166+0000 7f45db625640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d4142520 con 0x7f45d40a98f0 2024-03-25T18:04:04.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.166+0000 7f45db625640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d40b28a0 con 0x7f45d413f710 2024-03-25T18:04:04.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.166+0000 7f45db625640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d40b2a30 con 0x7f45d41430c0 2024-03-25T18:04:04.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.166+0000 7f45db625640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:04.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.166+0000 7f45db625640 10 monclient(hunting): authenticate will time out at 7372.745405s 2024-03-25T18:04:04.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.176+0000 7f45d8bc0640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f45d413f710 0x7f45d41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:52006/0 (socket says 172.21.15.18:52006) 2024-03-25T18:04:04.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.176+0000 7f45d8bc0640 1 -- 172.21.15.18:0/1473292039 learned_addr learned my addr 172.21.15.18:0/1473292039 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:04:04.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.192+0000 7f45cbfff640 1 -- 172.21.15.18:0/1473292039 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475634254 0 0) 0x7f45d40b28a0 con 0x7f45d413f710 2024-03-25T18:04:04.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.192+0000 7f45cbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:04:04.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.192+0000 7f45cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:04.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.192+0000 7f45cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:04.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.192+0000 7f45cbfff640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:04:04.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.192+0000 7f45cbfff640 1 -- 172.21.15.18:0/1473292039 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45c0003520 con 0x7f45d413f710 2024-03-25T18:04:04.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.203+0000 7f45cbfff640 1 -- 172.21.15.18:0/1473292039 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996875359 0 0) 0x7f45d40b2a30 con 0x7f45d41430c0 2024-03-25T18:04:04.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.203+0000 7f45cbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:04:04.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.203+0000 7f45cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:04.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.203+0000 7f45cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:04.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.203+0000 7f45cbfff640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:04:04.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.203+0000 7f45cbfff640 1 -- 172.21.15.18:0/1473292039 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45d40b28a0 con 0x7f45d41430c0 2024-03-25T18:04:04.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.210+0000 7f45cbfff640 1 -- 172.21.15.18:0/1473292039 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475068371 0 0) 0x7f45d4142520 con 0x7f45d40a98f0 2024-03-25T18:04:04.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.210+0000 7f45cbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:04:04.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.210+0000 7f45cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:04.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.210+0000 7f45cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:04.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.210+0000 7f45cbfff640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:04:04.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.210+0000 7f45cbfff640 1 -- 172.21.15.18:0/1473292039 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45d40b2a30 con 0x7f45d40a98f0 2024-03-25T18:04:04.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45cbfff640 1 -- 172.21.15.18:0/1473292039 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1180137907 0 0) 0x7f45d40b28a0 con 0x7f45d41430c0 2024-03-25T18:04:04.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45cbfff640 1 -- 172.21.15.18:0/1473292039 >> v1:172.21.15.169:6789/0 conn(0x7f45d413f710 legacy=0x7f45d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:04.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45cbfff640 1 -- 172.21.15.18:0/1473292039 >> v1:172.21.15.18:6789/0 conn(0x7f45d40a98f0 legacy=0x7f45d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:04.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45cbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:04:04.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45cbfff640 1 monclient: found mon.noname-b 2024-03-25T18:04:04.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:04.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45cbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:04:04.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45cbfff640 1 -- 172.21.15.18:0/1473292039 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d40b44c0 con 0x7f45d41430c0 2024-03-25T18:04:04.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45cbfff640 10 monclient: _finish_auth 0 2024-03-25T18:04:04.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45cbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:04:04.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:04.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45cbfff640 1 -- 172.21.15.18:0/1473292039 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f45cc0019f0 con 0x7f45d41430c0 2024-03-25T18:04:04.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45db625640 5 monclient: authenticate success, global_id 4580 2024-03-25T18:04:04.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45db625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:04.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:04.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45cbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:04:04.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45cbfff640 10 monclient: dump: 2024-03-25T18:04:04.216 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:04.216 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:04.216 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:04.216 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:04.216 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:04.216 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:04.216 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:04.216 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:04.216 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:04.216 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:04.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.212+0000 7f45db625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:04.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.216+0000 7f45cbfff640 1 -- 172.21.15.18:0/1473292039 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45cc001b50 con 0x7f45d41430c0 2024-03-25T18:04:04.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.216+0000 7f45cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:04.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.216+0000 7f45cbfff640 1 -- 172.21.15.18:0/1473292039 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f45cc001f60 con 0x7f45d41430c0 2024-03-25T18:04:04.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.216+0000 7f45cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:04.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.216+0000 7f45cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:04.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.216+0000 7f45cbfff640 10 monclient: dump: 2024-03-25T18:04:04.219 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:04.219 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:04.219 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:04.219 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:04.219 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:04.219 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:04.219 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:04.219 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:04.219 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:04.219 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:04.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.216+0000 7f45db625640 10 monclient: get_monmap_and_config success 2024-03-25T18:04:04.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.216+0000 7f45db625640 4 set_mon_vals no callback set 2024-03-25T18:04:04.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.217+0000 7f45db625640 10 monclient: shutdown 2024-03-25T18:04:04.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.217+0000 7f45db625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:04.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.217+0000 7f45db625640 1 -- 172.21.15.18:0/1473292039 >> v1:172.21.15.18:6790/0 conn(0x7f45d41430c0 legacy=0x7f45d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:04.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.217+0000 7f45db625640 1 -- 172.21.15.18:0/1473292039 shutdown_connections 2024-03-25T18:04:04.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.217+0000 7f45db625640 1 -- 172.21.15.18:0/1473292039 >> 172.21.15.18:0/1473292039 conn(0x7f45d40b7ea0 msgr2=0x7f45d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:04.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.218+0000 7f45db625640 1 -- 172.21.15.18:0/1473292039 shutdown_connections 2024-03-25T18:04:04.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.218+0000 7f45db625640 1 -- 172.21.15.18:0/1473292039 wait complete. 2024-03-25T18:04:04.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.219+0000 7f45db625640 10 monclient: build_initial_monmap 2024-03-25T18:04:04.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.219+0000 7f45db625640 10 monclient: monmap: 2024-03-25T18:04:04.223 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:04:04.223 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:04:04.223 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:04:04.223 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:04:04.223 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:04.223 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:04:04.223 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:04:04.223 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:04:04.223 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:04:04.223 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:04.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.219+0000 7f45db625640 1 librados: starting msgr at 2024-03-25T18:04:04.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.219+0000 7f45db625640 1 librados: starting objecter 2024-03-25T18:04:04.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.219+0000 7f45db625640 1 Processor -- start 2024-03-25T18:04:04.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.219+0000 7f45db625640 1 -- start start 2024-03-25T18:04:04.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.219+0000 7f45db625640 1 librados: setting wanted keys 2024-03-25T18:04:04.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.219+0000 7f45db625640 1 librados: calling monclient init 2024-03-25T18:04:04.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.219+0000 7f45db625640 10 monclient: init 2024-03-25T18:04:04.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.220+0000 7f45db625640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:04.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.220+0000 7f45db625640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:04:04.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.220+0000 7f45db625640 10 monclient(hunting): picked mon.noname-b con 0x7f45d40a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:04:04.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.220+0000 7f45db625640 10 monclient(hunting): picked mon.noname-a con 0x7f45d413f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:04:04.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.220+0000 7f45db625640 10 monclient(hunting): picked mon.noname-c con 0x7f45d41430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:04:04.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.220+0000 7f45db625640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d41397d0 con 0x7f45d413f710 2024-03-25T18:04:04.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.220+0000 7f45db625640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d413af80 con 0x7f45d40a98f0 2024-03-25T18:04:04.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.220+0000 7f45db625640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d41dba00 con 0x7f45d41430c0 2024-03-25T18:04:04.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.220+0000 7f45db625640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:04.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.220+0000 7f45db625640 10 monclient(hunting): authenticate will time out at 7372.799209s 2024-03-25T18:04:04.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.229+0000 7f45d8bc0640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f45d413f710 0x7f45d41dabe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:35868/0 (socket says 172.21.15.18:35868) 2024-03-25T18:04:04.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.229+0000 7f45d8bc0640 1 -- 172.21.15.18:0/2277121088 learned_addr learned my addr 172.21.15.18:0/2277121088 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:04:04.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.230+0000 7f45c9ffb640 10 client.?.objecter ms_handle_connect 0x7f45d413f710 2024-03-25T18:04:04.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.230+0000 7f45c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:04.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.231+0000 7f45c9ffb640 10 client.?.objecter ms_handle_connect 0x7f45d40a98f0 2024-03-25T18:04:04.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.231+0000 7f45c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:04.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.237+0000 7f45c9ffb640 10 client.?.objecter ms_handle_connect 0x7f45d41430c0 2024-03-25T18:04:04.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.237+0000 7f45c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:04.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.245+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264322522 0 0) 0x7f45d41397d0 con 0x7f45d413f710 2024-03-25T18:04:04.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.245+0000 7f45c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:04:04.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.245+0000 7f45c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:04.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.245+0000 7f45c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:04.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.245+0000 7f45c9ffb640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:04:04.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.245+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45b8004eb0 con 0x7f45d413f710 2024-03-25T18:04:04.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.249+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248000642 0 0) 0x7f45d413af80 con 0x7f45d40a98f0 2024-03-25T18:04:04.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.249+0000 7f45c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:04:04.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.249+0000 7f45c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:04.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.249+0000 7f45c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:04.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.249+0000 7f45c9ffb640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:04:04.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.249+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45d41397d0 con 0x7f45d40a98f0 2024-03-25T18:04:04.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.250+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686987149 0 0) 0x7f45d41dba00 con 0x7f45d41430c0 2024-03-25T18:04:04.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.250+0000 7f45c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:04:04.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.250+0000 7f45c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:04.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.250+0000 7f45c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:04.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.250+0000 7f45c9ffb640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:04:04.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.250+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45d413af80 con 0x7f45d41430c0 2024-03-25T18:04:04.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.259+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 563661545 0 0) 0x7f45b8004eb0 con 0x7f45d413f710 2024-03-25T18:04:04.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.259+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45d41dba00 con 0x7f45d413f710 2024-03-25T18:04:04.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.259+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f45c4001950 con 0x7f45d413f710 2024-03-25T18:04:04.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.259+0000 7f45c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:04.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.259+0000 7f45c9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:04:04.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.259+0000 7f45c9ffb640 10 monclient(hunting): dump: 2024-03-25T18:04:04.261 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:04.261 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:04.261 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:04.261 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:04.261 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:04.261 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:04.261 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:04.261 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:04.261 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:04.261 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:04.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.259+0000 7f45c9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:04:04.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 278100728 0 0) 0x7f45d41397d0 con 0x7f45d40a98f0 2024-03-25T18:04:04.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45b8004eb0 con 0x7f45d40a98f0 2024-03-25T18:04:04.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f45d0001870 con 0x7f45d40a98f0 2024-03-25T18:04:04.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:04.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45c9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:04.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45c9ffb640 10 monclient(hunting): dump: 2024-03-25T18:04:04.270 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:04.270 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:04.270 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:04.270 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:04.270 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:04.270 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:04.270 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:04.270 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:04.271 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:04.271 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:04.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4116036484 0 0) 0x7f45d41dba00 con 0x7f45d413f710 2024-03-25T18:04:04.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 >> v1:172.21.15.18:6790/0 conn(0x7f45d41430c0 legacy=0x7f45d41db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:04.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 >> v1:172.21.15.169:6789/0 conn(0x7f45d40a98f0 legacy=0x7f45d40b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:04.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45c9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:04:04.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45c9ffb640 1 monclient: found mon.a 2024-03-25T18:04:04.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:04.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45c9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:04:04.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d413b260 con 0x7f45d413f710 2024-03-25T18:04:04.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45c9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:04:04.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45c9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:04:04.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:04.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45db625640 5 monclient: authenticate success, global_id 4578 2024-03-25T18:04:04.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45db625640 10 monclient: _renew_subs 2024-03-25T18:04:04.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45db625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:04:04.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45db625640 1 -- 172.21.15.18:0/2277121088 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d40b74f0 con 0x7f45d413f710 2024-03-25T18:04:04.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45db625640 10 client.4578.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:04:04.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45db625640 10 monclient: _renew_subs 2024-03-25T18:04:04.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45db625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:04:04.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45db625640 1 -- 172.21.15.18:0/2277121088 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d412f7d0 con 0x7f45d413f710 2024-03-25T18:04:04.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.267+0000 7f45db625640 1 librados: init done 2024-03-25T18:04:04.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.269+0000 7f45db625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:04.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.269+0000 7f45db625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:04.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.269+0000 7f45db625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:04:04.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.269+0000 7f45db625640 1 -- 172.21.15.18:0/2277121088 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d40627b0 con 0x7f45d413f710 2024-03-25T18:04:04.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.594+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45c40047e0 con 0x7f45d413f710 2024-03-25T18:04:04.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.594+0000 7f45c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:04.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.594+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f45c4004af0 con 0x7f45d413f710 2024-03-25T18:04:04.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.594+0000 7f45c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:04.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.594+0000 7f45c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:04:04.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.594+0000 7f45c9ffb640 10 monclient: dump: 2024-03-25T18:04:04.596 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:04.596 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:04.596 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:04.597 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:04.597 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:04.597 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:04.597 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:04.597 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:04.597 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:04.597 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:04.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.594+0000 7f45cb7fe640 4 set_mon_vals no callback set 2024-03-25T18:04:04.598 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.595+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f45c401aac0 con 0x7f45d413f710 2024-03-25T18:04:04.598 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.595+0000 7f45d93c1640 10 monclient: get_auth_request con 0x7f45b806df20 auth_method 0 2024-03-25T18:04:04.598 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.596+0000 7f45c9ffb640 10 client.4578.objecter ms_handle_connect 0x7f45b806df20 2024-03-25T18:04:04.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.601+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f45c4054850 con 0x7f45d413f710 2024-03-25T18:04:04.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.601+0000 7f45c9ffb640 10 client.4578.objecter ms_dispatch 0x7f45d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:04:04.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.601+0000 7f45c9ffb640 3 client.4578.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:04:04.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.601+0000 7f45c9ffb640 3 client.4578.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:04:04.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.601+0000 7f45c9ffb640 20 client.4578.objecter dump_active .. 0 homeless 2024-03-25T18:04:04.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.601+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f45c4050360 con 0x7f45d413f710 2024-03-25T18:04:04.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.601+0000 7f45c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:04.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.601+0000 7f45c9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:04:04.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.856+0000 7f45db625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:04.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.857+0000 7f45db625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:04.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.857+0000 7f45db625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:04:04.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.857+0000 7f45db625640 1 -- 172.21.15.18:0/2277121088 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45d4139980 con 0x7f45d413f710 2024-03-25T18:04:04.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.887+0000 7f45c9ffb640 1 -- 172.21.15.18:0/2277121088 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f45d4139980 con 0x7f45d413f710 2024-03-25T18:04:04.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.887+0000 7f45c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:04.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.887+0000 7f45c9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:04:04.889 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:04:04.889 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:04:04.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.889+0000 7f45db625640 10 librados: watch_flush enter 2024-03-25T18:04:04.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.889+0000 7f45db625640 10 librados: watch_flush exit 2024-03-25T18:04:04.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.889+0000 7f45db625640 20 client.4578.objecter shutdown clearing up homeless session... 2024-03-25T18:04:04.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.889+0000 7f45db625640 10 client.4578.objecter successfully canceled tick 2024-03-25T18:04:04.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.889+0000 7f45db625640 1 -- 172.21.15.18:0/2277121088 >> v1:172.21.15.18:6800/3721244451 conn(0x7f45b806df20 legacy=0x7f45b80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:04.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.889+0000 7f45db625640 10 monclient: shutdown 2024-03-25T18:04:04.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.889+0000 7f45db625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:04.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.889+0000 7f45db625640 1 -- 172.21.15.18:0/2277121088 >> v1:172.21.15.18:6789/0 conn(0x7f45d413f710 legacy=0x7f45d41dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:04.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.889+0000 7f45db625640 1 -- 172.21.15.18:0/2277121088 shutdown_connections 2024-03-25T18:04:04.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.889+0000 7f45db625640 1 -- 172.21.15.18:0/2277121088 >> 172.21.15.18:0/2277121088 conn(0x7f45d40b7ea0 msgr2=0x7f45d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:04.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.889+0000 7f45db625640 1 -- 172.21.15.18:0/2277121088 shutdown_connections 2024-03-25T18:04:04.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.889+0000 7f45db625640 1 -- 172.21.15.18:0/2277121088 wait complete. 2024-03-25T18:04:04.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:04.889+0000 7f45db625640 1 librados: shutdown 2024-03-25T18:04:04.903 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:04:08.492 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:06.284 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:08.496 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:06.288 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:08.538 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:06.330 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:08.555 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:06.348 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:08.557 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:06.305 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:08.558 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:06.351 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:08.566 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:06.315 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:08.594 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:06.387 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:08.601 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:06.350 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:11.710 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:04:11.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.824+0000 7f32e5e2a640 10 monclient: get_monmap_and_config 2024-03-25T18:04:11.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.824+0000 7f32e5e2a640 10 monclient: build_initial_monmap 2024-03-25T18:04:11.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.825+0000 7f32e5e2a640 10 monclient: monmap: 2024-03-25T18:04:11.826 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:04:11.826 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:11.827 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:04:11.826367+0000 2024-03-25T18:04:11.827 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:04:11.826367+0000 2024-03-25T18:04:11.827 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:11.827 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:11.827 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:04:11.827 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:04:11.827 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:04:11.827 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:11.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.825+0000 7f32e5e2a640 1 Processor -- start 2024-03-25T18:04:11.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.825+0000 7f32e5e2a640 1 -- start start 2024-03-25T18:04:11.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.825+0000 7f32e5e2a640 10 monclient: init 2024-03-25T18:04:11.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.826+0000 7f32e5e2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:11.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.826+0000 7f32e5e2a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:04:11.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.826+0000 7f32e5e2a640 10 monclient(hunting): picked mon.noname-b con 0x7f32e00a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:04:11.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.826+0000 7f32e5e2a640 10 monclient(hunting): picked mon.noname-c con 0x7f32e013f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:04:11.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.826+0000 7f32e5e2a640 10 monclient(hunting): picked mon.noname-a con 0x7f32e01430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:04:11.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.826+0000 7f32e5e2a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e0142520 con 0x7f32e01430c0 2024-03-25T18:04:11.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.826+0000 7f32e5e2a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e00b28a0 con 0x7f32e013f710 2024-03-25T18:04:11.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.826+0000 7f32e5e2a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e00b2a30 con 0x7f32e00a98f0 2024-03-25T18:04:11.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.826+0000 7f32e5e2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:11.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.826+0000 7f32e5e2a640 10 monclient(hunting): authenticate will time out at 7380.405470s 2024-03-25T18:04:11.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.836+0000 7f32deffd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f32e013f710 0x7f32e01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:40154/0 (socket says 172.21.15.18:40154) 2024-03-25T18:04:11.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.836+0000 7f32deffd640 1 -- 172.21.15.18:0/1996087011 learned_addr learned my addr 172.21.15.18:0/1996087011 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:04:11.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.845+0000 7f32de7fc640 1 -- 172.21.15.18:0/1996087011 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691652387 0 0) 0x7f32e00b28a0 con 0x7f32e013f710 2024-03-25T18:04:11.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.845+0000 7f32de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:04:11.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.845+0000 7f32de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:11.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.845+0000 7f32de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:11.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.845+0000 7f32de7fc640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:04:11.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.845+0000 7f32de7fc640 1 -- 172.21.15.18:0/1996087011 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32c8003520 con 0x7f32e013f710 2024-03-25T18:04:11.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.849+0000 7f32de7fc640 1 -- 172.21.15.18:0/1996087011 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704682414 0 0) 0x7f32e00b2a30 con 0x7f32e00a98f0 2024-03-25T18:04:11.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.849+0000 7f32de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:04:11.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.849+0000 7f32de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:11.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.849+0000 7f32de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:11.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.849+0000 7f32de7fc640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:04:11.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.849+0000 7f32de7fc640 1 -- 172.21.15.18:0/1996087011 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e00b28a0 con 0x7f32e00a98f0 2024-03-25T18:04:11.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.854+0000 7f32de7fc640 1 -- 172.21.15.18:0/1996087011 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447886761 0 0) 0x7f32e0142520 con 0x7f32e01430c0 2024-03-25T18:04:11.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.854+0000 7f32de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:04:11.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.854+0000 7f32de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:11.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.854+0000 7f32de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:11.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.854+0000 7f32de7fc640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:04:11.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.854+0000 7f32de7fc640 1 -- 172.21.15.18:0/1996087011 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e00b2a30 con 0x7f32e01430c0 2024-03-25T18:04:11.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.859+0000 7f32de7fc640 1 -- 172.21.15.18:0/1996087011 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1057245790 0 0) 0x7f32e00b28a0 con 0x7f32e00a98f0 2024-03-25T18:04:11.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.859+0000 7f32de7fc640 1 -- 172.21.15.18:0/1996087011 >> v1:172.21.15.169:6789/0 conn(0x7f32e013f710 legacy=0x7f32e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:11.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.859+0000 7f32de7fc640 1 -- 172.21.15.18:0/1996087011 >> v1:172.21.15.18:6789/0 conn(0x7f32e01430c0 legacy=0x7f32e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:11.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.859+0000 7f32de7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:04:11.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.859+0000 7f32de7fc640 1 monclient: found mon.noname-b 2024-03-25T18:04:11.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.859+0000 7f32de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:11.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.859+0000 7f32de7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:04:11.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.859+0000 7f32de7fc640 1 -- 172.21.15.18:0/1996087011 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e00b44c0 con 0x7f32e00a98f0 2024-03-25T18:04:11.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.859+0000 7f32de7fc640 10 monclient: _finish_auth 0 2024-03-25T18:04:11.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.859+0000 7f32de7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:04:11.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.859+0000 7f32de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:11.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.859+0000 7f32de7fc640 1 -- 172.21.15.18:0/1996087011 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f32d0001670 con 0x7f32e00a98f0 2024-03-25T18:04:11.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.859+0000 7f32de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:11.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.859+0000 7f32de7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:04:11.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.859+0000 7f32de7fc640 10 monclient: dump: 2024-03-25T18:04:11.862 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:11.862 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:11.862 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:11.862 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:11.862 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:11.862 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:11.863 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:11.863 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:11.863 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:11.863 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:11.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.859+0000 7f32e5e2a640 5 monclient: authenticate success, global_id 4586 2024-03-25T18:04:11.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.859+0000 7f32e5e2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:11.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.873+0000 7f32de7fc640 1 -- 172.21.15.18:0/1996087011 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32d00017d0 con 0x7f32e00a98f0 2024-03-25T18:04:11.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.873+0000 7f32de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:11.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.873+0000 7f32de7fc640 1 -- 172.21.15.18:0/1996087011 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f32d0001be0 con 0x7f32e00a98f0 2024-03-25T18:04:11.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.873+0000 7f32de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:11.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.873+0000 7f32de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:11.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.873+0000 7f32de7fc640 10 monclient: dump: 2024-03-25T18:04:11.875 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:11.875 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:11.875 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:11.875 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:11.875 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:11.876 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:11.876 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:11.876 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:11.876 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:11.876 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:11.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.873+0000 7f32e5e2a640 10 monclient: get_monmap_and_config success 2024-03-25T18:04:11.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.873+0000 7f32e5e2a640 4 set_mon_vals no callback set 2024-03-25T18:04:11.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.874+0000 7f32e5e2a640 10 monclient: shutdown 2024-03-25T18:04:11.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.874+0000 7f32e5e2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:11.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.874+0000 7f32e5e2a640 1 -- 172.21.15.18:0/1996087011 >> v1:172.21.15.18:6790/0 conn(0x7f32e00a98f0 legacy=0x7f32e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:11.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.874+0000 7f32e5e2a640 1 -- 172.21.15.18:0/1996087011 shutdown_connections 2024-03-25T18:04:11.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.874+0000 7f32e5e2a640 1 -- 172.21.15.18:0/1996087011 >> 172.21.15.18:0/1996087011 conn(0x7f32e00b7ea0 msgr2=0x7f32e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:11.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.874+0000 7f32e5e2a640 1 -- 172.21.15.18:0/1996087011 shutdown_connections 2024-03-25T18:04:11.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.874+0000 7f32e5e2a640 1 -- 172.21.15.18:0/1996087011 wait complete. 2024-03-25T18:04:11.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.875+0000 7f32e5e2a640 10 monclient: build_initial_monmap 2024-03-25T18:04:11.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.875+0000 7f32e5e2a640 10 monclient: monmap: 2024-03-25T18:04:11.878 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:04:11.878 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:04:11.879 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:04:11.879 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:04:11.879 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:11.879 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:04:11.879 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:04:11.879 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:04:11.879 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:04:11.879 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:11.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.876+0000 7f32e5e2a640 1 librados: starting msgr at 2024-03-25T18:04:11.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.876+0000 7f32e5e2a640 1 librados: starting objecter 2024-03-25T18:04:11.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.876+0000 7f32e5e2a640 1 Processor -- start 2024-03-25T18:04:11.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.876+0000 7f32e5e2a640 1 -- start start 2024-03-25T18:04:11.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.876+0000 7f32e5e2a640 1 librados: setting wanted keys 2024-03-25T18:04:11.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.876+0000 7f32e5e2a640 1 librados: calling monclient init 2024-03-25T18:04:11.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.876+0000 7f32e5e2a640 10 monclient: init 2024-03-25T18:04:11.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.877+0000 7f32e5e2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:11.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.877+0000 7f32e5e2a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:04:11.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.877+0000 7f32e5e2a640 10 monclient(hunting): picked mon.noname-b con 0x7f32e00a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:04:11.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.877+0000 7f32e5e2a640 10 monclient(hunting): picked mon.noname-a con 0x7f32e013f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:04:11.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.877+0000 7f32e5e2a640 10 monclient(hunting): picked mon.noname-c con 0x7f32e01430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:04:11.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.877+0000 7f32e5e2a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e0136e60 con 0x7f32e013f710 2024-03-25T18:04:11.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.877+0000 7f32e5e2a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e01381a0 con 0x7f32e00a98f0 2024-03-25T18:04:11.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.877+0000 7f32e5e2a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e01370b0 con 0x7f32e01430c0 2024-03-25T18:04:11.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.877+0000 7f32e5e2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:11.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.877+0000 7f32e5e2a640 10 monclient(hunting): authenticate will time out at 7380.456485s 2024-03-25T18:04:11.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.887+0000 7f32df7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f32e00a98f0 0x7f32e0137a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:40178/0 (socket says 172.21.15.18:40178) 2024-03-25T18:04:11.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.887+0000 7f32df7fe640 1 -- 172.21.15.18:0/1116525308 learned_addr learned my addr 172.21.15.18:0/1116525308 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:04:11.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.888+0000 7f32c7fff640 10 client.?.objecter ms_handle_connect 0x7f32e00a98f0 2024-03-25T18:04:11.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.888+0000 7f32c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:11.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.900+0000 7f32c7fff640 10 client.?.objecter ms_handle_connect 0x7f32e01430c0 2024-03-25T18:04:11.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.900+0000 7f32c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:11.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.902+0000 7f32c7fff640 10 client.?.objecter ms_handle_connect 0x7f32e013f710 2024-03-25T18:04:11.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.902+0000 7f32c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:11.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.905+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123662670 0 0) 0x7f32e01381a0 con 0x7f32e00a98f0 2024-03-25T18:04:11.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.905+0000 7f32c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:04:11.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.905+0000 7f32c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:11.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.905+0000 7f32c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:11.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.905+0000 7f32c7fff640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:04:11.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.905+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32cc004eb0 con 0x7f32e00a98f0 2024-03-25T18:04:11.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.917+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1796735369 0 0) 0x7f32cc004eb0 con 0x7f32e00a98f0 2024-03-25T18:04:11.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.917+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32e01381a0 con 0x7f32e00a98f0 2024-03-25T18:04:11.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.917+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f32d0001670 con 0x7f32e00a98f0 2024-03-25T18:04:11.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.917+0000 7f32c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:11.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.917+0000 7f32c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:04:11.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.917+0000 7f32c7fff640 10 monclient(hunting): dump: 2024-03-25T18:04:11.920 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:11.920 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:11.920 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:11.920 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:11.920 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:11.920 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:11.920 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:11.920 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:11.920 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:11.920 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:11.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.917+0000 7f32c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:04:11.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.917+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71695585 0 0) 0x7f32e01370b0 con 0x7f32e01430c0 2024-03-25T18:04:11.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.917+0000 7f32c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:04:11.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.917+0000 7f32c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:11.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.917+0000 7f32c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:11.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.917+0000 7f32c7fff640 10 monclient(hunting): my global_id is 4589 2024-03-25T18:04:11.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.917+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32cc004eb0 con 0x7f32e01430c0 2024-03-25T18:04:11.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.919+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500260073 0 0) 0x7f32e0136e60 con 0x7f32e013f710 2024-03-25T18:04:11.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.919+0000 7f32c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:04:11.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.919+0000 7f32c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:11.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.919+0000 7f32c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:11.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.919+0000 7f32c7fff640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:04:11.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.919+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32e01370b0 con 0x7f32e013f710 2024-03-25T18:04:11.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.933+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2722906024 0 0) 0x7f32e01370b0 con 0x7f32e013f710 2024-03-25T18:04:11.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.933+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32e0136e60 con 0x7f32e013f710 2024-03-25T18:04:11.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.933+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f32d40017b0 con 0x7f32e013f710 2024-03-25T18:04:11.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.933+0000 7f32c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:11.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.933+0000 7f32c7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:04:11.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.933+0000 7f32c7fff640 10 monclient(hunting): dump: 2024-03-25T18:04:11.935 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:11.935 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:11.935 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:11.935 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:11.935 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:11.935 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:11.935 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:11.935 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:11.936 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:11.936 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:11.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 845973409 0 0) 0x7f32e01381a0 con 0x7f32e00a98f0 2024-03-25T18:04:11.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 >> v1:172.21.15.18:6790/0 conn(0x7f32e01430c0 legacy=0x7f32e013ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:11.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 >> v1:172.21.15.18:6789/0 conn(0x7f32e013f710 legacy=0x7f32e013a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:11.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32c7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:04:11.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32c7fff640 1 monclient: found mon.b 2024-03-25T18:04:11.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:11.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32c7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:11.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e0137390 con 0x7f32e00a98f0 2024-03-25T18:04:11.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32c7fff640 10 monclient: _finish_auth 0 2024-03-25T18:04:11.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32c7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:04:11.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:11.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32e5e2a640 5 monclient: authenticate success, global_id 4570 2024-03-25T18:04:11.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32e5e2a640 10 monclient: _renew_subs 2024-03-25T18:04:11.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32e5e2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:11.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32e5e2a640 1 -- 172.21.15.18:0/1116525308 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e01376d0 con 0x7f32e00a98f0 2024-03-25T18:04:11.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32e5e2a640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:04:11.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32e5e2a640 10 monclient: _renew_subs 2024-03-25T18:04:11.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32e5e2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:11.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32e5e2a640 1 -- 172.21.15.18:0/1116525308 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e0135530 con 0x7f32e00a98f0 2024-03-25T18:04:11.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.941+0000 7f32e5e2a640 1 librados: init done 2024-03-25T18:04:11.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.942+0000 7f32e5e2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:11.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.942+0000 7f32e5e2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:11.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.942+0000 7f32e5e2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:11.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:11.942+0000 7f32e5e2a640 1 -- 172.21.15.18:0/1116525308 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e00b3ef0 con 0x7f32e00a98f0 2024-03-25T18:04:12.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.217+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32d00035f0 con 0x7f32e00a98f0 2024-03-25T18:04:12.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.217+0000 7f32c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:12.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.217+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f32d0003be0 con 0x7f32e00a98f0 2024-03-25T18:04:12.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.217+0000 7f32c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:12.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.217+0000 7f32c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:12.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.217+0000 7f32c7fff640 10 monclient: dump: 2024-03-25T18:04:12.219 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:12.219 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:12.219 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:12.219 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:12.219 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:12.219 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:12.219 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:12.219 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:12.219 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:12.219 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:12.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.217+0000 7f32ddffb640 4 set_mon_vals no callback set 2024-03-25T18:04:12.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.218+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f32d0003dc0 con 0x7f32e00a98f0 2024-03-25T18:04:12.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.218+0000 7f32deffd640 10 monclient: get_auth_request con 0x7f32cc06e020 auth_method 0 2024-03-25T18:04:12.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.218+0000 7f32c7fff640 10 client.4570.objecter ms_handle_connect 0x7f32cc06e020 2024-03-25T18:04:12.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.223+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f32d00549a0 con 0x7f32e00a98f0 2024-03-25T18:04:12.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.223+0000 7f32c7fff640 10 client.4570.objecter ms_dispatch 0x7f32e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:04:12.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.223+0000 7f32c7fff640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:04:12.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.223+0000 7f32c7fff640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:04:12.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.223+0000 7f32c7fff640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-25T18:04:12.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.224+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f32d00502e0 con 0x7f32e00a98f0 2024-03-25T18:04:12.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.224+0000 7f32c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:12.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.224+0000 7f32c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:04:12.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.460+0000 7f32e5e2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:12.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.460+0000 7f32e5e2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:12.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.460+0000 7f32e5e2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:12.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.460+0000 7f32e5e2a640 1 -- 172.21.15.18:0/1116525308 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32e01357e0 con 0x7f32e00a98f0 2024-03-25T18:04:12.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.494+0000 7f32c7fff640 1 -- 172.21.15.18:0/1116525308 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f32d00523e0 con 0x7f32e00a98f0 2024-03-25T18:04:12.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.494+0000 7f32c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:12.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.494+0000 7f32c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:04:12.495 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:04:12.495 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:04:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.496+0000 7f32e5e2a640 10 librados: watch_flush enter 2024-03-25T18:04:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.496+0000 7f32e5e2a640 10 librados: watch_flush exit 2024-03-25T18:04:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.496+0000 7f32e5e2a640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-25T18:04:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.496+0000 7f32e5e2a640 10 client.4570.objecter successfully canceled tick 2024-03-25T18:04:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.496+0000 7f32e5e2a640 1 -- 172.21.15.18:0/1116525308 >> v1:172.21.15.18:6800/3721244451 conn(0x7f32cc06e020 legacy=0x7f32cc0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.496+0000 7f32e5e2a640 10 monclient: shutdown 2024-03-25T18:04:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.496+0000 7f32e5e2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.496+0000 7f32e5e2a640 1 -- 172.21.15.18:0/1116525308 >> v1:172.21.15.169:6789/0 conn(0x7f32e00a98f0 legacy=0x7f32e0137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:12.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.496+0000 7f32e5e2a640 1 -- 172.21.15.18:0/1116525308 shutdown_connections 2024-03-25T18:04:12.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.496+0000 7f32e5e2a640 1 -- 172.21.15.18:0/1116525308 >> 172.21.15.18:0/1116525308 conn(0x7f32e00b7ea0 msgr2=0x7f32e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:12.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.496+0000 7f32e5e2a640 1 -- 172.21.15.18:0/1116525308 shutdown_connections 2024-03-25T18:04:12.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.496+0000 7f32e5e2a640 1 -- 172.21.15.18:0/1116525308 wait complete. 2024-03-25T18:04:12.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:12.496+0000 7f32e5e2a640 1 librados: shutdown 2024-03-25T18:04:12.510 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:04:19.318 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:04:19.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.431+0000 7f7e161cf640 10 monclient: get_monmap_and_config 2024-03-25T18:04:19.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.431+0000 7f7e161cf640 10 monclient: build_initial_monmap 2024-03-25T18:04:19.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.431+0000 7f7e161cf640 10 monclient: monmap: 2024-03-25T18:04:19.432 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:04:19.432 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:19.432 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:04:19.431737+0000 2024-03-25T18:04:19.432 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:04:19.431737+0000 2024-03-25T18:04:19.432 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:19.432 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:19.432 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:04:19.432 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:04:19.432 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:04:19.432 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:19.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.432+0000 7f7e161cf640 1 Processor -- start 2024-03-25T18:04:19.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.432+0000 7f7e161cf640 1 -- start start 2024-03-25T18:04:19.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.432+0000 7f7e161cf640 10 monclient: init 2024-03-25T18:04:19.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.432+0000 7f7e161cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:19.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.432+0000 7f7e161cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:04:19.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.432+0000 7f7e161cf640 10 monclient(hunting): picked mon.noname-a con 0x7f7e100a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:04:19.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.432+0000 7f7e161cf640 10 monclient(hunting): picked mon.noname-b con 0x7f7e1013f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:04:19.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.432+0000 7f7e161cf640 10 monclient(hunting): picked mon.noname-c con 0x7f7e101430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:04:19.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.432+0000 7f7e161cf640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e10142520 con 0x7f7e100a98f0 2024-03-25T18:04:19.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.432+0000 7f7e161cf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e100b28a0 con 0x7f7e101430c0 2024-03-25T18:04:19.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.432+0000 7f7e161cf640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e100b2a30 con 0x7f7e1013f710 2024-03-25T18:04:19.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.432+0000 7f7e161cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:19.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.432+0000 7f7e161cf640 10 monclient(hunting): authenticate will time out at 7388.010615s 2024-03-25T18:04:19.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.439+0000 7f7e0f7fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f7e100a98f0 0x7f7e100a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:33160/0 (socket says 172.21.15.18:33160) 2024-03-25T18:04:19.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.439+0000 7f7e0f7fe640 1 -- 172.21.15.18:0/834945350 learned_addr learned my addr 172.21.15.18:0/834945350 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:04:19.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.453+0000 7f7e0e7fc640 1 -- 172.21.15.18:0/834945350 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855419732 0 0) 0x7f7e10142520 con 0x7f7e100a98f0 2024-03-25T18:04:19.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.453+0000 7f7e0e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:04:19.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.453+0000 7f7e0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:19.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.453+0000 7f7e0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:19.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.453+0000 7f7e0e7fc640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:04:19.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.453+0000 7f7e0e7fc640 1 -- 172.21.15.18:0/834945350 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df8003520 con 0x7f7e100a98f0 2024-03-25T18:04:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.461+0000 7f7e0e7fc640 1 -- 172.21.15.18:0/834945350 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 439048305 0 0) 0x7f7df8003520 con 0x7f7e100a98f0 2024-03-25T18:04:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.461+0000 7f7e0e7fc640 1 -- 172.21.15.18:0/834945350 >> v1:172.21.15.18:6790/0 conn(0x7f7e1013f710 legacy=0x7f7e101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:04:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.461+0000 7f7e0e7fc640 1 -- 172.21.15.18:0/834945350 >> v1:172.21.15.169:6789/0 conn(0x7f7e101430c0 legacy=0x7f7e1014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.461+0000 7f7e0e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:04:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.461+0000 7f7e0e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:04:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.461+0000 7f7e0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.461+0000 7f7e0e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:04:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.461+0000 7f7e0e7fc640 1 -- 172.21.15.18:0/834945350 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e100b44c0 con 0x7f7e100a98f0 2024-03-25T18:04:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.461+0000 7f7e0e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:04:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.461+0000 7f7e0e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:04:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.461+0000 7f7e0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:19.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.461+0000 7f7e0e7fc640 1 -- 172.21.15.18:0/834945350 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f7e00001690 con 0x7f7e100a98f0 2024-03-25T18:04:19.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.461+0000 7f7e0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:19.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.461+0000 7f7e0e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:04:19.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.461+0000 7f7e0e7fc640 10 monclient: dump: 2024-03-25T18:04:19.463 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:19.463 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:19.463 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:19.463 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:19.463 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:19.463 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:19.463 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:19.463 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:19.463 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:19.463 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:19.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.461+0000 7f7e161cf640 5 monclient: authenticate success, global_id 4587 2024-03-25T18:04:19.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.461+0000 7f7e161cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:19.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.472+0000 7f7e0e7fc640 1 -- 172.21.15.18:0/834945350 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e00001860 con 0x7f7e100a98f0 2024-03-25T18:04:19.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.472+0000 7f7e0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:19.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.472+0000 7f7e0e7fc640 1 -- 172.21.15.18:0/834945350 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f7e00001cb0 con 0x7f7e100a98f0 2024-03-25T18:04:19.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.472+0000 7f7e0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:19.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.472+0000 7f7e0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:04:19.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.472+0000 7f7e0e7fc640 10 monclient: dump: 2024-03-25T18:04:19.473 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:19.473 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:19.473 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:19.473 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:19.474 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:19.474 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:19.474 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:19.474 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:19.474 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:19.474 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:19.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.472+0000 7f7e161cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:04:19.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.472+0000 7f7e161cf640 4 set_mon_vals no callback set 2024-03-25T18:04:19.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.473+0000 7f7e161cf640 10 monclient: shutdown 2024-03-25T18:04:19.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.473+0000 7f7e161cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:19.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.473+0000 7f7e161cf640 1 -- 172.21.15.18:0/834945350 >> v1:172.21.15.18:6789/0 conn(0x7f7e100a98f0 legacy=0x7f7e100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:19.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.473+0000 7f7e161cf640 1 -- 172.21.15.18:0/834945350 shutdown_connections 2024-03-25T18:04:19.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.473+0000 7f7e161cf640 1 -- 172.21.15.18:0/834945350 >> 172.21.15.18:0/834945350 conn(0x7f7e100b7ea0 msgr2=0x7f7e100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:19.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.473+0000 7f7e161cf640 1 -- 172.21.15.18:0/834945350 shutdown_connections 2024-03-25T18:04:19.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.473+0000 7f7e161cf640 1 -- 172.21.15.18:0/834945350 wait complete. 2024-03-25T18:04:19.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.474+0000 7f7e161cf640 10 monclient: build_initial_monmap 2024-03-25T18:04:19.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.474+0000 7f7e161cf640 10 monclient: monmap: 2024-03-25T18:04:19.477 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:04:19.477 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:04:19.477 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:04:19.477 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:04:19.477 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:19.477 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:04:19.477 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:04:19.477 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:04:19.477 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:04:19.477 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:19.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.474+0000 7f7e161cf640 1 librados: starting msgr at 2024-03-25T18:04:19.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.474+0000 7f7e161cf640 1 librados: starting objecter 2024-03-25T18:04:19.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.474+0000 7f7e161cf640 1 Processor -- start 2024-03-25T18:04:19.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.474+0000 7f7e161cf640 1 -- start start 2024-03-25T18:04:19.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.474+0000 7f7e161cf640 1 librados: setting wanted keys 2024-03-25T18:04:19.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.474+0000 7f7e161cf640 1 librados: calling monclient init 2024-03-25T18:04:19.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.474+0000 7f7e161cf640 10 monclient: init 2024-03-25T18:04:19.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.475+0000 7f7e161cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:19.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.475+0000 7f7e161cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:04:19.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.475+0000 7f7e161cf640 10 monclient(hunting): picked mon.noname-a con 0x7f7e100a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:04:19.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.475+0000 7f7e161cf640 10 monclient(hunting): picked mon.noname-b con 0x7f7e1013f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:04:19.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.475+0000 7f7e161cf640 10 monclient(hunting): picked mon.noname-c con 0x7f7e101430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:04:19.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.475+0000 7f7e161cf640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e10136f30 con 0x7f7e100a98f0 2024-03-25T18:04:19.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.475+0000 7f7e161cf640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e10135570 con 0x7f7e1013f710 2024-03-25T18:04:19.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.475+0000 7f7e161cf640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e10135070 con 0x7f7e101430c0 2024-03-25T18:04:19.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.475+0000 7f7e161cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:19.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.475+0000 7f7e161cf640 10 monclient(hunting): authenticate will time out at 7388.053271s 2024-03-25T18:04:19.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.491+0000 7f7e0effd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f7e1013f710 0x7f7e101dade0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:36420/0 (socket says 172.21.15.18:36420) 2024-03-25T18:04:19.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.491+0000 7f7e0effd640 1 -- 172.21.15.18:0/3009799030 learned_addr learned my addr 172.21.15.18:0/3009799030 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:04:19.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.495+0000 7f7df7fff640 10 client.?.objecter ms_handle_connect 0x7f7e1013f710 2024-03-25T18:04:19.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.495+0000 7f7df7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:19.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.497+0000 7f7df7fff640 10 client.?.objecter ms_handle_connect 0x7f7e101430c0 2024-03-25T18:04:19.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.497+0000 7f7df7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:19.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.502+0000 7f7df7fff640 10 client.?.objecter ms_handle_connect 0x7f7e100a98f0 2024-03-25T18:04:19.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.502+0000 7f7df7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:19.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.507+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176500011 0 0) 0x7f7e10135570 con 0x7f7e1013f710 2024-03-25T18:04:19.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.507+0000 7f7df7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:04:19.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.507+0000 7f7df7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:19.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.507+0000 7f7df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:19.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.507+0000 7f7df7fff640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:04:19.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.507+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dfc004eb0 con 0x7f7e1013f710 2024-03-25T18:04:19.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.509+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076910549 0 0) 0x7f7e10135070 con 0x7f7e101430c0 2024-03-25T18:04:19.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.509+0000 7f7df7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:04:19.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.509+0000 7f7df7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:19.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.509+0000 7f7df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:19.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.509+0000 7f7df7fff640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:04:19.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.509+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e10135570 con 0x7f7e101430c0 2024-03-25T18:04:19.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.510+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709233832 0 0) 0x7f7e10136f30 con 0x7f7e100a98f0 2024-03-25T18:04:19.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.511+0000 7f7df7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:04:19.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.511+0000 7f7df7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:19.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.511+0000 7f7df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:19.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.511+0000 7f7df7fff640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:04:19.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.511+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e10135070 con 0x7f7e100a98f0 2024-03-25T18:04:19.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.520+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2238266869 0 0) 0x7f7dfc004eb0 con 0x7f7e1013f710 2024-03-25T18:04:19.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.520+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e10136f30 con 0x7f7e1013f710 2024-03-25T18:04:19.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.520+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f7e04001960 con 0x7f7e1013f710 2024-03-25T18:04:19.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.520+0000 7f7df7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:19.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.520+0000 7f7df7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:04:19.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.520+0000 7f7df7fff640 10 monclient(hunting): dump: 2024-03-25T18:04:19.520 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:19.521 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:19.521 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:19.521 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:19.521 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:19.521 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:19.521 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:19.521 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:19.521 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:19.521 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:19.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.520+0000 7f7df7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:04:19.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.530+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1729785456 0 0) 0x7f7e10135570 con 0x7f7e101430c0 2024-03-25T18:04:19.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.530+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dfc004eb0 con 0x7f7e101430c0 2024-03-25T18:04:19.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.530+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f7e08001b10 con 0x7f7e101430c0 2024-03-25T18:04:19.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.530+0000 7f7df7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:19.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.530+0000 7f7df7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:19.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.530+0000 7f7df7fff640 10 monclient(hunting): dump: 2024-03-25T18:04:19.530 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:19.530 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:19.531 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:19.531 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:19.531 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:19.531 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:19.531 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:19.531 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:19.531 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:19.531 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:19.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2291375674 0 0) 0x7f7dfc004eb0 con 0x7f7e101430c0 2024-03-25T18:04:19.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 >> v1:172.21.15.169:6789/0 conn(0x7f7e1013f710 legacy=0x7f7e101dade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:19.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 >> v1:172.21.15.18:6789/0 conn(0x7f7e100a98f0 legacy=0x7f7e101d4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:19.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7df7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:04:19.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7df7fff640 1 monclient: found mon.c 2024-03-25T18:04:19.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7df7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:19.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7df7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:04:19.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e101d41e0 con 0x7f7e101430c0 2024-03-25T18:04:19.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7df7fff640 10 monclient: _finish_auth 0 2024-03-25T18:04:19.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7df7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:04:19.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7df7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:19.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7e161cf640 5 monclient: authenticate success, global_id 4592 2024-03-25T18:04:19.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7e161cf640 10 monclient: _renew_subs 2024-03-25T18:04:19.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7e161cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:04:19.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7e161cf640 1 -- 172.21.15.18:0/3009799030 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e10135cf0 con 0x7f7e101430c0 2024-03-25T18:04:19.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7e161cf640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:04:19.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7e161cf640 10 monclient: _renew_subs 2024-03-25T18:04:19.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7e161cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:04:19.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7e161cf640 1 -- 172.21.15.18:0/3009799030 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1013ca50 con 0x7f7e101430c0 2024-03-25T18:04:19.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.534+0000 7f7e161cf640 1 librados: init done 2024-03-25T18:04:19.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.535+0000 7f7e161cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:19.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.535+0000 7f7e161cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:19.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.535+0000 7f7e161cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:04:19.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.535+0000 7f7e161cf640 1 -- 172.21.15.18:0/3009799030 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e101d43d0 con 0x7f7e101430c0 2024-03-25T18:04:19.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.537+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e08002000 con 0x7f7e101430c0 2024-03-25T18:04:19.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.537+0000 7f7df7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:19.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.537+0000 7f7e0dffb640 4 set_mon_vals no callback set 2024-03-25T18:04:19.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.854+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f7e08003b30 con 0x7f7e101430c0 2024-03-25T18:04:19.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.854+0000 7f7df7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:19.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.854+0000 7f7df7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:19.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.854+0000 7f7df7fff640 10 monclient: dump: 2024-03-25T18:04:19.854 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:19.855 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:19.855 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:19.855 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:19.855 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:19.855 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:19.855 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:19.855 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:19.855 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:19.855 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:19.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.854+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f7e08019b00 con 0x7f7e101430c0 2024-03-25T18:04:19.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.855+0000 7f7e0f7fe640 10 monclient: get_auth_request con 0x7f7dfc06df70 auth_method 0 2024-03-25T18:04:19.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.855+0000 7f7df7fff640 10 client.4592.objecter ms_handle_connect 0x7f7dfc06df70 2024-03-25T18:04:19.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.864+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f7e08053810 con 0x7f7e101430c0 2024-03-25T18:04:19.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.864+0000 7f7df7fff640 10 client.4592.objecter ms_dispatch 0x7f7e10000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:04:19.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.864+0000 7f7df7fff640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:04:19.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.864+0000 7f7df7fff640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:04:19.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.864+0000 7f7df7fff640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-25T18:04:19.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.864+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e0804f150 con 0x7f7e101430c0 2024-03-25T18:04:19.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.864+0000 7f7df7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:19.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:19.864+0000 7f7df7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:04:20.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.122+0000 7f7e161cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:20.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.122+0000 7f7e161cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:20.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.122+0000 7f7e161cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:04:20.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.122+0000 7f7e161cf640 1 -- 172.21.15.18:0/3009799030 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e100b3ef0 con 0x7f7e101430c0 2024-03-25T18:04:20.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.240+0000 7f7df7fff640 1 -- 172.21.15.18:0/3009799030 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f7e08051250 con 0x7f7e101430c0 2024-03-25T18:04:20.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.240+0000 7f7df7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:20.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.240+0000 7f7df7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:04:20.241 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:04:20.241 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:04:20.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.242+0000 7f7e161cf640 10 librados: watch_flush enter 2024-03-25T18:04:20.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.243+0000 7f7e161cf640 10 librados: watch_flush exit 2024-03-25T18:04:20.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.243+0000 7f7e161cf640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-25T18:04:20.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.243+0000 7f7e161cf640 10 client.4592.objecter successfully canceled tick 2024-03-25T18:04:20.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.243+0000 7f7e161cf640 1 -- 172.21.15.18:0/3009799030 >> v1:172.21.15.18:6800/3721244451 conn(0x7f7dfc06df70 legacy=0x7f7dfc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:20.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.243+0000 7f7e161cf640 10 monclient: shutdown 2024-03-25T18:04:20.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.243+0000 7f7e161cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:20.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.243+0000 7f7e161cf640 1 -- 172.21.15.18:0/3009799030 >> v1:172.21.15.18:6790/0 conn(0x7f7e101430c0 legacy=0x7f7e101db4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:20.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.243+0000 7f7e161cf640 1 -- 172.21.15.18:0/3009799030 shutdown_connections 2024-03-25T18:04:20.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.243+0000 7f7e161cf640 1 -- 172.21.15.18:0/3009799030 >> 172.21.15.18:0/3009799030 conn(0x7f7e100b7ea0 msgr2=0x7f7e10074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:20.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.243+0000 7f7e161cf640 1 -- 172.21.15.18:0/3009799030 shutdown_connections 2024-03-25T18:04:20.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.243+0000 7f7e161cf640 1 -- 172.21.15.18:0/3009799030 wait complete. 2024-03-25T18:04:20.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:20.243+0000 7f7e161cf640 1 librados: shutdown 2024-03-25T18:04:20.254 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:04:27.061 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:04:27.161 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.160+0000 7f8c7b8d3640 10 monclient: get_monmap_and_config 2024-03-25T18:04:27.161 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.160+0000 7f8c7b8d3640 10 monclient: build_initial_monmap 2024-03-25T18:04:27.161 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.160+0000 7f8c7b8d3640 10 monclient: monmap: 2024-03-25T18:04:27.161 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:04:27.161 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:27.161 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:04:27.161116+0000 2024-03-25T18:04:27.161 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:04:27.161116+0000 2024-03-25T18:04:27.162 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:27.162 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:27.162 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:04:27.162 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:04:27.162 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:04:27.162 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:27.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.161+0000 7f8c7b8d3640 1 Processor -- start 2024-03-25T18:04:27.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.161+0000 7f8c7b8d3640 1 -- start start 2024-03-25T18:04:27.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.161+0000 7f8c7b8d3640 10 monclient: init 2024-03-25T18:04:27.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.162+0000 7f8c7b8d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:27.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.162+0000 7f8c7b8d3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:04:27.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.162+0000 7f8c7b8d3640 10 monclient(hunting): picked mon.noname-a con 0x7f8c740705b0 addr v1:172.21.15.18:6789/0 2024-03-25T18:04:27.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.162+0000 7f8c7b8d3640 10 monclient(hunting): picked mon.noname-b con 0x7f8c741402d0 addr v1:172.21.15.18:6790/0 2024-03-25T18:04:27.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.162+0000 7f8c7b8d3640 10 monclient(hunting): picked mon.noname-c con 0x7f8c74149ae0 addr v1:172.21.15.169:6789/0 2024-03-25T18:04:27.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.162+0000 7f8c7b8d3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c740718c0 con 0x7f8c740705b0 2024-03-25T18:04:27.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.162+0000 7f8c7b8d3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c740714c0 con 0x7f8c74149ae0 2024-03-25T18:04:27.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.162+0000 7f8c7b8d3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c74072b50 con 0x7f8c741402d0 2024-03-25T18:04:27.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.162+0000 7f8c7b8d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:27.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.162+0000 7f8c7b8d3640 10 monclient(hunting): authenticate will time out at 7395.740376s 2024-03-25T18:04:27.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.169+0000 7f8c7966f640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f8c740705b0 0x7f8c7413d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:41210/0 (socket says 172.21.15.18:41210) 2024-03-25T18:04:27.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.169+0000 7f8c7966f640 1 -- 172.21.15.18:0/662338561 learned_addr learned my addr 172.21.15.18:0/662338561 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:04:27.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.173+0000 7f8c6bfff640 1 -- 172.21.15.18:0/662338561 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152724449 0 0) 0x7f8c740718c0 con 0x7f8c740705b0 2024-03-25T18:04:27.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.173+0000 7f8c6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:04:27.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.173+0000 7f8c6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:27.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.173+0000 7f8c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:27.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.173+0000 7f8c6bfff640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:04:27.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.173+0000 7f8c6bfff640 1 -- 172.21.15.18:0/662338561 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c60003520 con 0x7f8c740705b0 2024-03-25T18:04:27.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.181+0000 7f8c6bfff640 1 -- 172.21.15.18:0/662338561 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410000526 0 0) 0x7f8c740714c0 con 0x7f8c74149ae0 2024-03-25T18:04:27.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.181+0000 7f8c6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:04:27.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.181+0000 7f8c6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:27.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.181+0000 7f8c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:27.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.181+0000 7f8c6bfff640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:04:27.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.181+0000 7f8c6bfff640 1 -- 172.21.15.18:0/662338561 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c740718c0 con 0x7f8c74149ae0 2024-03-25T18:04:27.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.183+0000 7f8c6bfff640 1 -- 172.21.15.18:0/662338561 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3704288964 0 0) 0x7f8c60003520 con 0x7f8c740705b0 2024-03-25T18:04:27.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.183+0000 7f8c6bfff640 1 -- 172.21.15.18:0/662338561 >> v1:172.21.15.18:6790/0 conn(0x7f8c741402d0 legacy=0x7f8c74140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:04:27.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.183+0000 7f8c6bfff640 1 -- 172.21.15.18:0/662338561 >> v1:172.21.15.169:6789/0 conn(0x7f8c74149ae0 legacy=0x7f8c7414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:27.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.183+0000 7f8c6bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:04:27.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.183+0000 7f8c6bfff640 1 monclient: found mon.noname-a 2024-03-25T18:04:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.183+0000 7f8c6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.183+0000 7f8c6bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:04:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.183+0000 7f8c6bfff640 1 -- 172.21.15.18:0/662338561 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7413dd10 con 0x7f8c740705b0 2024-03-25T18:04:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.183+0000 7f8c6bfff640 10 monclient: _finish_auth 0 2024-03-25T18:04:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.183+0000 7f8c6bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:04:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.183+0000 7f8c6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.183+0000 7f8c7b8d3640 5 monclient: authenticate success, global_id 4593 2024-03-25T18:04:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.183+0000 7f8c6bfff640 1 -- 172.21.15.18:0/662338561 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8c70001690 con 0x7f8c740705b0 2024-03-25T18:04:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.183+0000 7f8c6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.183+0000 7f8c6bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:04:27.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.183+0000 7f8c6bfff640 10 monclient: dump: 2024-03-25T18:04:27.186 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:27.186 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:27.186 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:27.186 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:27.186 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:27.186 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:27.186 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:27.186 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:27.186 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:27.187 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:27.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.183+0000 7f8c7b8d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:27.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.205+0000 7f8c6bfff640 1 -- 172.21.15.18:0/662338561 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c700017f0 con 0x7f8c740705b0 2024-03-25T18:04:27.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.205+0000 7f8c6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:27.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.205+0000 7f8c6bfff640 1 -- 172.21.15.18:0/662338561 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8c70001c00 con 0x7f8c740705b0 2024-03-25T18:04:27.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.205+0000 7f8c6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:27.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.205+0000 7f8c6bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:04:27.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.205+0000 7f8c6bfff640 10 monclient: dump: 2024-03-25T18:04:27.207 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:27.207 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:27.207 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:27.207 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:27.207 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:27.207 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:27.208 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:27.208 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:27.208 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:27.208 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:27.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.205+0000 7f8c7b8d3640 10 monclient: get_monmap_and_config success 2024-03-25T18:04:27.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.205+0000 7f8c7b8d3640 4 set_mon_vals no callback set 2024-03-25T18:04:27.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.206+0000 7f8c7b8d3640 10 monclient: shutdown 2024-03-25T18:04:27.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.206+0000 7f8c7b8d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:27.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.206+0000 7f8c7b8d3640 1 -- 172.21.15.18:0/662338561 >> v1:172.21.15.18:6789/0 conn(0x7f8c740705b0 legacy=0x7f8c7413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:27.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.206+0000 7f8c7b8d3640 1 -- 172.21.15.18:0/662338561 shutdown_connections 2024-03-25T18:04:27.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.206+0000 7f8c7b8d3640 1 -- 172.21.15.18:0/662338561 >> 172.21.15.18:0/662338561 conn(0x7f8c740b5d10 msgr2=0x7f8c740b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:27.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.206+0000 7f8c7b8d3640 1 -- 172.21.15.18:0/662338561 shutdown_connections 2024-03-25T18:04:27.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.206+0000 7f8c7b8d3640 1 -- 172.21.15.18:0/662338561 wait complete. 2024-03-25T18:04:27.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.207+0000 7f8c7b8d3640 10 monclient: build_initial_monmap 2024-03-25T18:04:27.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.207+0000 7f8c7b8d3640 10 monclient: monmap: 2024-03-25T18:04:27.212 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:04:27.212 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:04:27.212 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:04:27.212 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:04:27.212 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:27.212 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:04:27.212 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:04:27.212 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:04:27.212 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:04:27.213 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:27.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.208+0000 7f8c7b8d3640 1 librados: starting msgr at 2024-03-25T18:04:27.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.208+0000 7f8c7b8d3640 1 librados: starting objecter 2024-03-25T18:04:27.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.208+0000 7f8c7b8d3640 1 Processor -- start 2024-03-25T18:04:27.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.208+0000 7f8c7b8d3640 1 -- start start 2024-03-25T18:04:27.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.208+0000 7f8c7b8d3640 1 librados: setting wanted keys 2024-03-25T18:04:27.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.208+0000 7f8c7b8d3640 1 librados: calling monclient init 2024-03-25T18:04:27.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.208+0000 7f8c7b8d3640 10 monclient: init 2024-03-25T18:04:27.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.208+0000 7f8c7b8d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:27.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.208+0000 7f8c7b8d3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:04:27.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.209+0000 7f8c7b8d3640 10 monclient(hunting): picked mon.noname-c con 0x7f8c740705b0 addr v1:172.21.15.18:6790/0 2024-03-25T18:04:27.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.209+0000 7f8c7b8d3640 10 monclient(hunting): picked mon.noname-b con 0x7f8c741402d0 addr v1:172.21.15.169:6789/0 2024-03-25T18:04:27.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.209+0000 7f8c7b8d3640 10 monclient(hunting): picked mon.noname-a con 0x7f8c74149ae0 addr v1:172.21.15.18:6789/0 2024-03-25T18:04:27.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.209+0000 7f8c7b8d3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c74143ca0 con 0x7f8c74149ae0 2024-03-25T18:04:27.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.209+0000 7f8c7b8d3640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7413dfd0 con 0x7f8c741402d0 2024-03-25T18:04:27.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.209+0000 7f8c7b8d3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c741443a0 con 0x7f8c740705b0 2024-03-25T18:04:27.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.209+0000 7f8c7b8d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:27.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.209+0000 7f8c7b8d3640 10 monclient(hunting): authenticate will time out at 7395.786848s 2024-03-25T18:04:27.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.214+0000 7f8c7966f640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f8c740705b0 0x7f8c741d8f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:48522/0 (socket says 172.21.15.18:48522) 2024-03-25T18:04:27.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.214+0000 7f8c7966f640 1 -- 172.21.15.18:0/665692136 learned_addr learned my addr 172.21.15.18:0/665692136 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:04:27.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.215+0000 7f8c69ffb640 10 client.?.objecter ms_handle_connect 0x7f8c740705b0 2024-03-25T18:04:27.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.215+0000 7f8c69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:27.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.217+0000 7f8c69ffb640 10 client.?.objecter ms_handle_connect 0x7f8c741402d0 2024-03-25T18:04:27.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.217+0000 7f8c69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:27.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.218+0000 7f8c69ffb640 10 client.?.objecter ms_handle_connect 0x7f8c74149ae0 2024-03-25T18:04:27.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.218+0000 7f8c69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:27.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.230+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161974636 0 0) 0x7f8c7413dfd0 con 0x7f8c741402d0 2024-03-25T18:04:27.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.230+0000 7f8c69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:04:27.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.230+0000 7f8c69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:27.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.230+0000 7f8c69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:27.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.230+0000 7f8c69ffb640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:04:27.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.230+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c58004eb0 con 0x7f8c741402d0 2024-03-25T18:04:27.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.233+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647798890 0 0) 0x7f8c741443a0 con 0x7f8c740705b0 2024-03-25T18:04:27.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.233+0000 7f8c69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:04:27.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.233+0000 7f8c69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:27.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.233+0000 7f8c69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:27.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.233+0000 7f8c69ffb640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:04:27.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.233+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c7413dfd0 con 0x7f8c740705b0 2024-03-25T18:04:27.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.239+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616529798 0 0) 0x7f8c74143ca0 con 0x7f8c74149ae0 2024-03-25T18:04:27.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.239+0000 7f8c69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:04:27.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.239+0000 7f8c69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:27.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.239+0000 7f8c69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:27.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.239+0000 7f8c69ffb640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:04:27.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.239+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c741443a0 con 0x7f8c74149ae0 2024-03-25T18:04:27.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.248+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 338592424 0 0) 0x7f8c58004eb0 con 0x7f8c741402d0 2024-03-25T18:04:27.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.248+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c74143ca0 con 0x7f8c741402d0 2024-03-25T18:04:27.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.248+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8c64001870 con 0x7f8c741402d0 2024-03-25T18:04:27.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.248+0000 7f8c69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:27.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.248+0000 7f8c69ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:04:27.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.248+0000 7f8c69ffb640 10 monclient(hunting): dump: 2024-03-25T18:04:27.249 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:27.249 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:27.249 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:27.249 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:27.249 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:27.250 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:27.250 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:27.250 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:27.250 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:27.250 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:27.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.248+0000 7f8c69ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:04:27.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.262+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4230660237 0 0) 0x7f8c741443a0 con 0x7f8c74149ae0 2024-03-25T18:04:27.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.262+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c58004eb0 con 0x7f8c74149ae0 2024-03-25T18:04:27.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.262+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8c6c001bd0 con 0x7f8c74149ae0 2024-03-25T18:04:27.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.262+0000 7f8c69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:27.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.262+0000 7f8c69ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:04:27.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.262+0000 7f8c69ffb640 10 monclient(hunting): dump: 2024-03-25T18:04:27.263 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:27.263 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:27.263 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:27.263 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:27.263 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:27.264 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:27.264 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:27.264 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:27.264 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:27.264 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:27.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.270+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 627153369 0 0) 0x7f8c7413dfd0 con 0x7f8c740705b0 2024-03-25T18:04:27.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c741443a0 con 0x7f8c740705b0 2024-03-25T18:04:27.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8c70001b40 con 0x7f8c740705b0 2024-03-25T18:04:27.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:27.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c69ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:27.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c69ffb640 10 monclient(hunting): dump: 2024-03-25T18:04:27.273 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:27.273 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:27.274 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:27.274 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:27.274 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:27.274 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:27.274 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:27.274 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:27.274 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:27.274 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:27.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4193194309 0 0) 0x7f8c74143ca0 con 0x7f8c741402d0 2024-03-25T18:04:27.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 >> v1:172.21.15.18:6790/0 conn(0x7f8c740705b0 legacy=0x7f8c741d8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:27.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 >> v1:172.21.15.18:6789/0 conn(0x7f8c74149ae0 legacy=0x7f8c741e0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:27.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c69ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:04:27.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c69ffb640 1 monclient: found mon.b 2024-03-25T18:04:27.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:27.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c69ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:27.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c74144530 con 0x7f8c741402d0 2024-03-25T18:04:27.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c69ffb640 10 monclient: _finish_auth 0 2024-03-25T18:04:27.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c69ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:04:27.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:27.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c7b8d3640 5 monclient: authenticate success, global_id 4582 2024-03-25T18:04:27.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c7b8d3640 10 monclient: _renew_subs 2024-03-25T18:04:27.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c7b8d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:27.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c7b8d3640 1 -- 172.21.15.18:0/665692136 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c741d9690 con 0x7f8c741402d0 2024-03-25T18:04:27.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c7b8d3640 10 client.4582.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:04:27.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c7b8d3640 10 monclient: _renew_subs 2024-03-25T18:04:27.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c7b8d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:27.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c7b8d3640 1 -- 172.21.15.18:0/665692136 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c741d99e0 con 0x7f8c741402d0 2024-03-25T18:04:27.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.271+0000 7f8c7b8d3640 1 librados: init done 2024-03-25T18:04:27.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.272+0000 7f8c7b8d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:27.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.272+0000 7f8c7b8d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:27.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.272+0000 7f8c7b8d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:27.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.272+0000 7f8c7b8d3640 1 -- 172.21.15.18:0/665692136 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c740627b0 con 0x7f8c741402d0 2024-03-25T18:04:27.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.560+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c640035f0 con 0x7f8c741402d0 2024-03-25T18:04:27.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.560+0000 7f8c69ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:27.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.560+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f8c64001d80 con 0x7f8c741402d0 2024-03-25T18:04:27.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.560+0000 7f8c69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:27.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.560+0000 7f8c69ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:27.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.560+0000 7f8c69ffb640 10 monclient: dump: 2024-03-25T18:04:27.561 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:27.561 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:27.561 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:27.561 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:27.562 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:27.562 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:27.562 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:27.562 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:27.562 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:27.562 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:27.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.560+0000 7f8c6b7fe640 4 set_mon_vals no callback set 2024-03-25T18:04:27.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.561+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f8c640197b0 con 0x7f8c741402d0 2024-03-25T18:04:27.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.562+0000 7f8c7966f640 10 monclient: get_auth_request con 0x7f8c5806e0f0 auth_method 0 2024-03-25T18:04:27.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.562+0000 7f8c69ffb640 10 client.4582.objecter ms_handle_connect 0x7f8c5806e0f0 2024-03-25T18:04:27.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.565+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f8c64053560 con 0x7f8c741402d0 2024-03-25T18:04:27.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.565+0000 7f8c69ffb640 10 client.4582.objecter ms_dispatch 0x7f8c74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:04:27.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.565+0000 7f8c69ffb640 3 client.4582.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:04:27.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.565+0000 7f8c69ffb640 3 client.4582.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:04:27.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.565+0000 7f8c69ffb640 20 client.4582.objecter dump_active .. 0 homeless 2024-03-25T18:04:27.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.566+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c6404eea0 con 0x7f8c741402d0 2024-03-25T18:04:27.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.566+0000 7f8c69ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:27.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.566+0000 7f8c69ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:04:27.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.821+0000 7f8c7b8d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:27.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.821+0000 7f8c7b8d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:27.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.821+0000 7f8c7b8d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:27.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.821+0000 7f8c7b8d3640 1 -- 172.21.15.18:0/665692136 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c741407a0 con 0x7f8c741402d0 2024-03-25T18:04:27.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.880+0000 7f8c69ffb640 1 -- 172.21.15.18:0/665692136 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f8c64050fa0 con 0x7f8c741402d0 2024-03-25T18:04:27.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.880+0000 7f8c69ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:27.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.880+0000 7f8c69ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:04:27.881 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:04:27.881 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:04:27.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.882+0000 7f8c7b8d3640 10 librados: watch_flush enter 2024-03-25T18:04:27.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.882+0000 7f8c7b8d3640 10 librados: watch_flush exit 2024-03-25T18:04:27.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.882+0000 7f8c7b8d3640 20 client.4582.objecter shutdown clearing up homeless session... 2024-03-25T18:04:27.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.882+0000 7f8c7b8d3640 10 client.4582.objecter successfully canceled tick 2024-03-25T18:04:27.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.882+0000 7f8c7b8d3640 1 -- 172.21.15.18:0/665692136 >> v1:172.21.15.18:6800/3721244451 conn(0x7f8c5806e0f0 legacy=0x7f8c580705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:27.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.882+0000 7f8c7b8d3640 10 monclient: shutdown 2024-03-25T18:04:27.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.882+0000 7f8c7b8d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:27.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.882+0000 7f8c7b8d3640 1 -- 172.21.15.18:0/665692136 >> v1:172.21.15.169:6789/0 conn(0x7f8c741402d0 legacy=0x7f8c74142290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:27.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.882+0000 7f8c7b8d3640 1 -- 172.21.15.18:0/665692136 shutdown_connections 2024-03-25T18:04:27.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.882+0000 7f8c7b8d3640 1 -- 172.21.15.18:0/665692136 >> 172.21.15.18:0/665692136 conn(0x7f8c740b5d10 msgr2=0x7f8c7413b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:27.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.882+0000 7f8c7b8d3640 1 -- 172.21.15.18:0/665692136 shutdown_connections 2024-03-25T18:04:27.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.882+0000 7f8c7b8d3640 1 -- 172.21.15.18:0/665692136 wait complete. 2024-03-25T18:04:27.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:27.882+0000 7f8c7b8d3640 1 librados: shutdown 2024-03-25T18:04:27.894 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:04:28.706 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:26.499 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:28.717 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:26.510 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:28.756 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:26.548 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:28.769 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:26.562 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:28.778 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:26.570 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:28.781 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:26.530 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:28.786 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:26.535 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:28.804 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:26.597 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:28.814 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:26.563 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:34.701 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:04:34.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.814+0000 7f5898063640 10 monclient: get_monmap_and_config 2024-03-25T18:04:34.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.814+0000 7f5898063640 10 monclient: build_initial_monmap 2024-03-25T18:04:34.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.814+0000 7f5898063640 10 monclient: monmap: 2024-03-25T18:04:34.815 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:04:34.815 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:34.815 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:04:34.814621+0000 2024-03-25T18:04:34.815 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:04:34.814621+0000 2024-03-25T18:04:34.815 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:34.815 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:34.815 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:04:34.815 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:04:34.816 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:04:34.816 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:34.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.815+0000 7f5898063640 1 Processor -- start 2024-03-25T18:04:34.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.815+0000 7f5898063640 1 -- start start 2024-03-25T18:04:34.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.815+0000 7f5898063640 10 monclient: init 2024-03-25T18:04:34.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.816+0000 7f5898063640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:34.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.816+0000 7f5898063640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:04:34.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.816+0000 7f5898063640 10 monclient(hunting): picked mon.noname-c con 0x7f58900a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:04:34.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.816+0000 7f5898063640 10 monclient(hunting): picked mon.noname-b con 0x7f589013f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:04:34.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.816+0000 7f5898063640 10 monclient(hunting): picked mon.noname-a con 0x7f58901430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:04:34.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.816+0000 7f5898063640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5890142520 con 0x7f58901430c0 2024-03-25T18:04:34.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.816+0000 7f5898063640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58900b28a0 con 0x7f58900a98f0 2024-03-25T18:04:34.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.816+0000 7f5898063640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58900b2a30 con 0x7f589013f710 2024-03-25T18:04:34.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.816+0000 7f5898063640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:34.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.816+0000 7f5898063640 10 monclient(hunting): authenticate will time out at 7403.394052s 2024-03-25T18:04:34.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.822+0000 7f58955fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f589013f710 0x7f58901429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:42202/0 (socket says 172.21.15.18:42202) 2024-03-25T18:04:34.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.822+0000 7f58955fe640 1 -- 172.21.15.18:0/1464158719 learned_addr learned my addr 172.21.15.18:0/1464158719 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:04:34.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.825+0000 7f5894dfd640 1 -- 172.21.15.18:0/1464158719 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2795813271 0 0) 0x7f58900b2a30 con 0x7f589013f710 2024-03-25T18:04:34.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.825+0000 7f5894dfd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:34.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.825+0000 7f5894dfd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:34.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.825+0000 7f5894dfd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:34.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.825+0000 7f5894dfd640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:04:34.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.825+0000 7f5894dfd640 1 -- 172.21.15.18:0/1464158719 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f587c003520 con 0x7f589013f710 2024-03-25T18:04:34.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5894dfd640 1 -- 172.21.15.18:0/1464158719 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3450520988 0 0) 0x7f587c003520 con 0x7f589013f710 2024-03-25T18:04:34.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5894dfd640 1 -- 172.21.15.18:0/1464158719 >> v1:172.21.15.169:6789/0 conn(0x7f58900a98f0 legacy=0x7f58900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:34.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5894dfd640 1 -- 172.21.15.18:0/1464158719 >> v1:172.21.15.18:6789/0 conn(0x7f58901430c0 legacy=0x7f589014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:04:34.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5894dfd640 10 monclient: _finish_hunting 0 2024-03-25T18:04:34.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5894dfd640 1 monclient: found mon.noname-b 2024-03-25T18:04:34.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5894dfd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:34.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5894dfd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:04:34.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5894dfd640 1 -- 172.21.15.18:0/1464158719 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58900b44c0 con 0x7f589013f710 2024-03-25T18:04:34.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5894dfd640 10 monclient: _finish_auth 0 2024-03-25T18:04:34.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5894dfd640 10 monclient: _check_auth_tickets 2024-03-25T18:04:34.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5894dfd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:34.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5894dfd640 1 -- 172.21.15.18:0/1464158719 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5880001690 con 0x7f589013f710 2024-03-25T18:04:34.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5898063640 5 monclient: authenticate success, global_id 4598 2024-03-25T18:04:34.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5898063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:34.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5894dfd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:34.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5894dfd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:04:34.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5894dfd640 10 monclient: dump: 2024-03-25T18:04:34.835 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:34.835 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:34.835 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:34.835 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:34.835 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:34.835 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:34.835 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:34.836 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:34.836 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:34.836 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:34.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.833+0000 7f5898063640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:34.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.837+0000 7f5894dfd640 1 -- 172.21.15.18:0/1464158719 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5880001860 con 0x7f589013f710 2024-03-25T18:04:34.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.837+0000 7f5894dfd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:34.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.837+0000 7f5894dfd640 1 -- 172.21.15.18:0/1464158719 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5880001cb0 con 0x7f589013f710 2024-03-25T18:04:34.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.837+0000 7f5898063640 10 monclient: get_monmap_and_config success 2024-03-25T18:04:34.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.837+0000 7f5898063640 4 set_mon_vals no callback set 2024-03-25T18:04:34.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.837+0000 7f5894dfd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:34.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.837+0000 7f5894dfd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:34.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.837+0000 7f5894dfd640 10 monclient: dump: 2024-03-25T18:04:34.838 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:34.838 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:34.839 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:34.839 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:34.839 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:34.839 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:34.839 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:34.839 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:34.839 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:34.839 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:34.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.838+0000 7f5898063640 10 monclient: shutdown 2024-03-25T18:04:34.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.838+0000 7f5898063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:34.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.838+0000 7f5898063640 1 -- 172.21.15.18:0/1464158719 >> v1:172.21.15.18:6790/0 conn(0x7f589013f710 legacy=0x7f58901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:34.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.838+0000 7f5898063640 1 -- 172.21.15.18:0/1464158719 shutdown_connections 2024-03-25T18:04:34.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.838+0000 7f5898063640 1 -- 172.21.15.18:0/1464158719 >> 172.21.15.18:0/1464158719 conn(0x7f58900b7ea0 msgr2=0x7f58900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:34.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.839+0000 7f5898063640 1 -- 172.21.15.18:0/1464158719 shutdown_connections 2024-03-25T18:04:34.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.839+0000 7f5898063640 1 -- 172.21.15.18:0/1464158719 wait complete. 2024-03-25T18:04:34.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.839+0000 7f5898063640 10 monclient: build_initial_monmap 2024-03-25T18:04:34.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.839+0000 7f5898063640 10 monclient: monmap: 2024-03-25T18:04:34.842 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:04:34.842 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:04:34.842 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:04:34.842 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:04:34.842 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:34.843 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:04:34.843 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:04:34.843 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:04:34.843 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:04:34.843 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:34.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.840+0000 7f5898063640 1 librados: starting msgr at 2024-03-25T18:04:34.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.840+0000 7f5898063640 1 librados: starting objecter 2024-03-25T18:04:34.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.840+0000 7f5898063640 1 Processor -- start 2024-03-25T18:04:34.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.840+0000 7f5898063640 1 -- start start 2024-03-25T18:04:34.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.840+0000 7f5898063640 1 librados: setting wanted keys 2024-03-25T18:04:34.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.840+0000 7f5898063640 1 librados: calling monclient init 2024-03-25T18:04:34.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.840+0000 7f5898063640 10 monclient: init 2024-03-25T18:04:34.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.840+0000 7f5898063640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:34.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.840+0000 7f5898063640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:04:34.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.840+0000 7f5898063640 10 monclient(hunting): picked mon.noname-c con 0x7f58900a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:04:34.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.840+0000 7f5898063640 10 monclient(hunting): picked mon.noname-a con 0x7f589013f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:04:34.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.840+0000 7f5898063640 10 monclient(hunting): picked mon.noname-b con 0x7f58901430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:04:34.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.840+0000 7f5898063640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58900b7890 con 0x7f589013f710 2024-03-25T18:04:34.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.840+0000 7f5898063640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589012d970 con 0x7f58901430c0 2024-03-25T18:04:34.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.840+0000 7f5898063640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589013ba50 con 0x7f58900a98f0 2024-03-25T18:04:34.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.840+0000 7f5898063640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:34.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.840+0000 7f5898063640 10 monclient(hunting): authenticate will time out at 7403.418521s 2024-03-25T18:04:34.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.846+0000 7f5896600640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f58901430c0 0x7f589012d260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:50058/0 (socket says 172.21.15.18:50058) 2024-03-25T18:04:34.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.846+0000 7f5896600640 1 -- 172.21.15.18:0/650695226 learned_addr learned my addr 172.21.15.18:0/650695226 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:04:34.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.857+0000 7f58867fc640 10 client.?.objecter ms_handle_connect 0x7f589013f710 2024-03-25T18:04:34.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.857+0000 7f58867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:34.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.858+0000 7f58867fc640 10 client.?.objecter ms_handle_connect 0x7f58901430c0 2024-03-25T18:04:34.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.858+0000 7f58867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:34.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.867+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577486065 0 0) 0x7f58900b7890 con 0x7f589013f710 2024-03-25T18:04:34.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.867+0000 7f58867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:04:34.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.867+0000 7f58867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:34.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.867+0000 7f58867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:34.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.867+0000 7f58867fc640 10 monclient(hunting): my global_id is 4599 2024-03-25T18:04:34.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.868+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5874004eb0 con 0x7f589013f710 2024-03-25T18:04:34.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.868+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405219324 0 0) 0x7f589012d970 con 0x7f58901430c0 2024-03-25T18:04:34.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.868+0000 7f58867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:04:34.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.868+0000 7f58867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:34.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.868+0000 7f58867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:34.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.868+0000 7f58867fc640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:04:34.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.868+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58900b7890 con 0x7f58901430c0 2024-03-25T18:04:34.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.869+0000 7f58867fc640 10 client.?.objecter ms_handle_connect 0x7f58900a98f0 2024-03-25T18:04:34.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.869+0000 7f58867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:34.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.875+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3373288760 0 0) 0x7f58900b7890 con 0x7f58901430c0 2024-03-25T18:04:34.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.875+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f589012d970 con 0x7f58901430c0 2024-03-25T18:04:34.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.875+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5888001a90 con 0x7f58901430c0 2024-03-25T18:04:34.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.875+0000 7f58867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:34.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.875+0000 7f58867fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:04:34.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.875+0000 7f58867fc640 10 monclient(hunting): dump: 2024-03-25T18:04:34.876 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:34.876 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:34.877 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:34.877 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:34.877 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:34.877 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:34.877 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:34.877 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:34.877 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:34.877 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:34.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.875+0000 7f58867fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:04:34.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.885+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337588534 0 0) 0x7f589013ba50 con 0x7f58900a98f0 2024-03-25T18:04:34.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.885+0000 7f58867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:04:34.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.885+0000 7f58867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:34.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.885+0000 7f58867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:34.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.885+0000 7f58867fc640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:04:34.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.885+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58900b7890 con 0x7f58900a98f0 2024-03-25T18:04:34.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 983101520 0 0) 0x7f589012d970 con 0x7f58901430c0 2024-03-25T18:04:34.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 >> v1:172.21.15.18:6790/0 conn(0x7f58900a98f0 legacy=0x7f589013c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:34.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 >> v1:172.21.15.18:6789/0 conn(0x7f589013f710 legacy=0x7f589012cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:34.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f58867fc640 10 monclient: _finish_hunting 0 2024-03-25T18:04:34.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f58867fc640 1 monclient: found mon.b 2024-03-25T18:04:34.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f58867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:34.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f58867fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:34.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589013bd30 con 0x7f58901430c0 2024-03-25T18:04:34.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f58867fc640 10 monclient: _finish_auth 0 2024-03-25T18:04:34.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f58867fc640 10 monclient: _check_auth_tickets 2024-03-25T18:04:34.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f58867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:34.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f5898063640 5 monclient: authenticate success, global_id 4585 2024-03-25T18:04:34.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f5898063640 10 monclient: _renew_subs 2024-03-25T18:04:34.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f5898063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:34.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f5898063640 1 -- 172.21.15.18:0/650695226 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58901399a0 con 0x7f58901430c0 2024-03-25T18:04:34.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f5898063640 10 client.4585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:04:34.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f5898063640 10 monclient: _renew_subs 2024-03-25T18:04:34.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f5898063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:34.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f5898063640 1 -- 172.21.15.18:0/650695226 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f589012fe20 con 0x7f58901430c0 2024-03-25T18:04:34.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.887+0000 7f5898063640 1 librados: init done 2024-03-25T18:04:34.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.888+0000 7f5898063640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:34.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.888+0000 7f5898063640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:34.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.888+0000 7f5898063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:34.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.888+0000 7f5898063640 1 -- 172.21.15.18:0/650695226 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f589014f2d0 con 0x7f58901430c0 2024-03-25T18:04:34.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.891+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5888001fa0 con 0x7f58901430c0 2024-03-25T18:04:34.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.891+0000 7f58867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:34.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:34.891+0000 7f5887fff640 4 set_mon_vals no callback set 2024-03-25T18:04:35.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.168+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5888004d80 con 0x7f58901430c0 2024-03-25T18:04:35.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.168+0000 7f58867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:35.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.168+0000 7f58867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:35.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.168+0000 7f58867fc640 10 monclient: dump: 2024-03-25T18:04:35.169 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:35.169 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:35.169 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:35.169 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:35.170 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:35.170 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:35.170 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:35.170 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:35.170 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:35.170 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:35.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.169+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f588801ad50 con 0x7f58901430c0 2024-03-25T18:04:35.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.170+0000 7f5895dff640 10 monclient: get_auth_request con 0x7f587406ddc0 auth_method 0 2024-03-25T18:04:35.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.170+0000 7f58867fc640 10 client.4585.objecter ms_handle_connect 0x7f587406ddc0 2024-03-25T18:04:35.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.172+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f5888054ab0 con 0x7f58901430c0 2024-03-25T18:04:35.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.172+0000 7f58867fc640 10 client.4585.objecter ms_dispatch 0x7f5890000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:04:35.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.172+0000 7f58867fc640 3 client.4585.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:04:35.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.172+0000 7f58867fc640 3 client.4585.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:04:35.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.172+0000 7f58867fc640 20 client.4585.objecter dump_active .. 0 homeless 2024-03-25T18:04:35.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.174+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f58880503f0 con 0x7f58901430c0 2024-03-25T18:04:35.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.175+0000 7f58867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:35.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.175+0000 7f58867fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:04:35.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.402+0000 7f5898063640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:35.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.402+0000 7f5898063640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:35.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.402+0000 7f5898063640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:35.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.402+0000 7f5898063640 1 -- 172.21.15.18:0/650695226 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58900627b0 con 0x7f58901430c0 2024-03-25T18:04:35.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.564+0000 7f58867fc640 1 -- 172.21.15.18:0/650695226 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f58880524f0 con 0x7f58901430c0 2024-03-25T18:04:35.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.564+0000 7f58867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:35.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.564+0000 7f58867fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:04:35.565 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:04:35.565 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:04:35.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.566+0000 7f5898063640 10 librados: watch_flush enter 2024-03-25T18:04:35.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.566+0000 7f5898063640 10 librados: watch_flush exit 2024-03-25T18:04:35.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.566+0000 7f5898063640 20 client.4585.objecter shutdown clearing up homeless session... 2024-03-25T18:04:35.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.566+0000 7f5898063640 10 client.4585.objecter successfully canceled tick 2024-03-25T18:04:35.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.566+0000 7f5898063640 1 -- 172.21.15.18:0/650695226 >> v1:172.21.15.18:6800/3721244451 conn(0x7f587406ddc0 legacy=0x7f5874070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:35.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.566+0000 7f5898063640 10 monclient: shutdown 2024-03-25T18:04:35.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.566+0000 7f5898063640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:35.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.566+0000 7f5898063640 1 -- 172.21.15.18:0/650695226 >> v1:172.21.15.169:6789/0 conn(0x7f58901430c0 legacy=0x7f589012d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:35.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.566+0000 7f5898063640 1 -- 172.21.15.18:0/650695226 shutdown_connections 2024-03-25T18:04:35.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.566+0000 7f5898063640 1 -- 172.21.15.18:0/650695226 >> 172.21.15.18:0/650695226 conn(0x7f58900b7ea0 msgr2=0x7f5890074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:35.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.566+0000 7f5898063640 1 -- 172.21.15.18:0/650695226 shutdown_connections 2024-03-25T18:04:35.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.566+0000 7f5898063640 1 -- 172.21.15.18:0/650695226 wait complete. 2024-03-25T18:04:35.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:35.566+0000 7f5898063640 1 librados: shutdown 2024-03-25T18:04:35.580 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:04:42.387 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:04:42.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.502+0000 7f3b423b5640 10 monclient: get_monmap_and_config 2024-03-25T18:04:42.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.502+0000 7f3b423b5640 10 monclient: build_initial_monmap 2024-03-25T18:04:42.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.502+0000 7f3b423b5640 10 monclient: monmap: 2024-03-25T18:04:42.503 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:04:42.503 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:42.503 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:04:42.503203+0000 2024-03-25T18:04:42.503 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:04:42.503203+0000 2024-03-25T18:04:42.503 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:42.504 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:42.504 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:04:42.504 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:04:42.504 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:04:42.504 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:42.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.503+0000 7f3b423b5640 1 Processor -- start 2024-03-25T18:04:42.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.503+0000 7f3b423b5640 1 -- start start 2024-03-25T18:04:42.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.503+0000 7f3b423b5640 10 monclient: init 2024-03-25T18:04:42.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.504+0000 7f3b423b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:42.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.504+0000 7f3b423b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:04:42.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.504+0000 7f3b423b5640 10 monclient(hunting): picked mon.noname-a con 0x7f3b3c0b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T18:04:42.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.504+0000 7f3b423b5640 10 monclient(hunting): picked mon.noname-b con 0x7f3b3c0b1730 addr v1:172.21.15.18:6790/0 2024-03-25T18:04:42.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.504+0000 7f3b423b5640 10 monclient(hunting): picked mon.noname-c con 0x7f3b3c14be90 addr v1:172.21.15.169:6789/0 2024-03-25T18:04:42.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.504+0000 7f3b423b5640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c0af940 con 0x7f3b3c0b4d60 2024-03-25T18:04:42.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.504+0000 7f3b423b5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c0b0a70 con 0x7f3b3c14be90 2024-03-25T18:04:42.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.504+0000 7f3b423b5640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c0b0d50 con 0x7f3b3c0b1730 2024-03-25T18:04:42.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.504+0000 7f3b423b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:42.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.504+0000 7f3b423b5640 10 monclient(hunting): authenticate will time out at 7411.082049s 2024-03-25T18:04:42.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.511+0000 7f3b3b7fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f3b3c0b1730 0x7f3b3c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:47020/0 (socket says 172.21.15.18:47020) 2024-03-25T18:04:42.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.511+0000 7f3b3b7fe640 1 -- 172.21.15.18:0/2225251870 learned_addr learned my addr 172.21.15.18:0/2225251870 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:04:42.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.524+0000 7f3b3affd640 1 -- 172.21.15.18:0/2225251870 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296156668 0 0) 0x7f3b3c0b0d50 con 0x7f3b3c0b1730 2024-03-25T18:04:42.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.524+0000 7f3b3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:42.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.524+0000 7f3b3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:42.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.524+0000 7f3b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:42.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.524+0000 7f3b3affd640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:04:42.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.524+0000 7f3b3affd640 1 -- 172.21.15.18:0/2225251870 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b24003520 con 0x7f3b3c0b1730 2024-03-25T18:04:42.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.536+0000 7f3b3affd640 1 -- 172.21.15.18:0/2225251870 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150108191 0 0) 0x7f3b3c0af940 con 0x7f3b3c0b4d60 2024-03-25T18:04:42.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.536+0000 7f3b3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:42.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.536+0000 7f3b3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:42.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.536+0000 7f3b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:42.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.536+0000 7f3b3affd640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:04:42.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.536+0000 7f3b3affd640 1 -- 172.21.15.18:0/2225251870 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b3c0b0d50 con 0x7f3b3c0b4d60 2024-03-25T18:04:42.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.537+0000 7f3b3affd640 1 -- 172.21.15.18:0/2225251870 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284506264 0 0) 0x7f3b3c0b0a70 con 0x7f3b3c14be90 2024-03-25T18:04:42.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.537+0000 7f3b3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:42.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.537+0000 7f3b3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:42.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.537+0000 7f3b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:42.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.537+0000 7f3b3affd640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:04:42.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.537+0000 7f3b3affd640 1 -- 172.21.15.18:0/2225251870 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b3c0af940 con 0x7f3b3c14be90 2024-03-25T18:04:42.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b3affd640 1 -- 172.21.15.18:0/2225251870 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 476806820 0 0) 0x7f3b24003520 con 0x7f3b3c0b1730 2024-03-25T18:04:42.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b3affd640 1 -- 172.21.15.18:0/2225251870 >> v1:172.21.15.169:6789/0 conn(0x7f3b3c14be90 legacy=0x7f3b3c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:42.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b3affd640 1 -- 172.21.15.18:0/2225251870 >> v1:172.21.15.18:6789/0 conn(0x7f3b3c0b4d60 legacy=0x7f3b3c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:42.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b3affd640 10 monclient: _finish_hunting 0 2024-03-25T18:04:42.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b3affd640 1 monclient: found mon.noname-b 2024-03-25T18:04:42.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:42.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b3affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:04:42.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b3affd640 1 -- 172.21.15.18:0/2225251870 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3c0b1030 con 0x7f3b3c0b1730 2024-03-25T18:04:42.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b3affd640 10 monclient: _finish_auth 0 2024-03-25T18:04:42.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b3affd640 10 monclient: _check_auth_tickets 2024-03-25T18:04:42.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:42.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b3affd640 1 -- 172.21.15.18:0/2225251870 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3b30001690 con 0x7f3b3c0b1730 2024-03-25T18:04:42.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b423b5640 5 monclient: authenticate success, global_id 4604 2024-03-25T18:04:42.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b423b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:42.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:42.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b3affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:04:42.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b3affd640 10 monclient: dump: 2024-03-25T18:04:42.549 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:42.549 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:42.549 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:42.549 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:42.549 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:42.549 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:42.549 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:42.549 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:42.549 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:42.549 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:42.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.547+0000 7f3b423b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:42.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.576+0000 7f3b3affd640 1 -- 172.21.15.18:0/2225251870 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b300017f0 con 0x7f3b3c0b1730 2024-03-25T18:04:42.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.576+0000 7f3b3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:42.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.576+0000 7f3b3affd640 1 -- 172.21.15.18:0/2225251870 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3b30001c00 con 0x7f3b3c0b1730 2024-03-25T18:04:42.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.576+0000 7f3b423b5640 10 monclient: get_monmap_and_config success 2024-03-25T18:04:42.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.576+0000 7f3b423b5640 4 set_mon_vals no callback set 2024-03-25T18:04:42.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.576+0000 7f3b3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:42.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.576+0000 7f3b3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:42.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.576+0000 7f3b3affd640 10 monclient: dump: 2024-03-25T18:04:42.577 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:42.577 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:42.577 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:42.577 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:42.577 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:42.577 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:42.577 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:42.577 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:42.577 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:42.577 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:42.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.576+0000 7f3b423b5640 10 monclient: shutdown 2024-03-25T18:04:42.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.576+0000 7f3b423b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:42.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.576+0000 7f3b423b5640 1 -- 172.21.15.18:0/2225251870 >> v1:172.21.15.18:6790/0 conn(0x7f3b3c0b1730 legacy=0x7f3b3c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:42.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.576+0000 7f3b423b5640 1 -- 172.21.15.18:0/2225251870 shutdown_connections 2024-03-25T18:04:42.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.576+0000 7f3b423b5640 1 -- 172.21.15.18:0/2225251870 >> 172.21.15.18:0/2225251870 conn(0x7f3b3c0617a0 msgr2=0x7f3b3c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:42.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.577+0000 7f3b423b5640 1 -- 172.21.15.18:0/2225251870 shutdown_connections 2024-03-25T18:04:42.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.577+0000 7f3b423b5640 1 -- 172.21.15.18:0/2225251870 wait complete. 2024-03-25T18:04:42.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.577+0000 7f3b423b5640 10 monclient: build_initial_monmap 2024-03-25T18:04:42.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.577+0000 7f3b423b5640 10 monclient: monmap: 2024-03-25T18:04:42.580 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:04:42.580 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:04:42.580 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:04:42.580 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:04:42.580 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:42.581 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:04:42.581 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:04:42.581 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:04:42.581 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:04:42.581 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:42.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.577+0000 7f3b423b5640 1 librados: starting msgr at 2024-03-25T18:04:42.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.577+0000 7f3b423b5640 1 librados: starting objecter 2024-03-25T18:04:42.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.578+0000 7f3b423b5640 1 Processor -- start 2024-03-25T18:04:42.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.578+0000 7f3b423b5640 1 -- start start 2024-03-25T18:04:42.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.578+0000 7f3b423b5640 1 librados: setting wanted keys 2024-03-25T18:04:42.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.578+0000 7f3b423b5640 1 librados: calling monclient init 2024-03-25T18:04:42.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.578+0000 7f3b423b5640 10 monclient: init 2024-03-25T18:04:42.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.578+0000 7f3b423b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:42.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.578+0000 7f3b423b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:04:42.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.578+0000 7f3b423b5640 10 monclient(hunting): picked mon.noname-b con 0x7f3b3c0b1730 addr v1:172.21.15.169:6789/0 2024-03-25T18:04:42.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.578+0000 7f3b423b5640 10 monclient(hunting): picked mon.noname-c con 0x7f3b3c0b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T18:04:42.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.578+0000 7f3b423b5640 10 monclient(hunting): picked mon.noname-a con 0x7f3b3c14be90 addr v1:172.21.15.18:6789/0 2024-03-25T18:04:42.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.578+0000 7f3b423b5640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c138250 con 0x7f3b3c14be90 2024-03-25T18:04:42.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.578+0000 7f3b423b5640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c1383e0 con 0x7f3b3c0b1730 2024-03-25T18:04:42.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.578+0000 7f3b423b5640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c1387a0 con 0x7f3b3c0b4d60 2024-03-25T18:04:42.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.578+0000 7f3b423b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:42.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.578+0000 7f3b423b5640 10 monclient(hunting): authenticate will time out at 7411.156528s 2024-03-25T18:04:42.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.585+0000 7f3b3b7fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f3b3c0b4d60 0x7f3b3c1db260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:47064/0 (socket says 172.21.15.18:47064) 2024-03-25T18:04:42.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.585+0000 7f3b3b7fe640 1 -- 172.21.15.18:0/1996374310 learned_addr learned my addr 172.21.15.18:0/1996374310 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:04:42.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.586+0000 7f3b38ff9640 10 client.?.objecter ms_handle_connect 0x7f3b3c0b4d60 2024-03-25T18:04:42.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.586+0000 7f3b38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:42.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.597+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660768402 0 0) 0x7f3b3c1387a0 con 0x7f3b3c0b4d60 2024-03-25T18:04:42.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.597+0000 7f3b38ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:04:42.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.597+0000 7f3b38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:42.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.597+0000 7f3b38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:42.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.597+0000 7f3b38ff9640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:04:42.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.597+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b28004eb0 con 0x7f3b3c0b4d60 2024-03-25T18:04:42.598 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.597+0000 7f3b38ff9640 10 client.?.objecter ms_handle_connect 0x7f3b3c14be90 2024-03-25T18:04:42.598 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.597+0000 7f3b38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:42.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.600+0000 7f3b38ff9640 10 client.?.objecter ms_handle_connect 0x7f3b3c0b1730 2024-03-25T18:04:42.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.600+0000 7f3b38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:42.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.620+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887333546 0 0) 0x7f3b3c138250 con 0x7f3b3c14be90 2024-03-25T18:04:42.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.620+0000 7f3b38ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:04:42.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.620+0000 7f3b38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:42.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.620+0000 7f3b38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:42.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.620+0000 7f3b38ff9640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:04:42.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.620+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b3c1387a0 con 0x7f3b3c14be90 2024-03-25T18:04:42.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.620+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197910681 0 0) 0x7f3b3c1383e0 con 0x7f3b3c0b1730 2024-03-25T18:04:42.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.620+0000 7f3b38ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:04:42.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.620+0000 7f3b38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:42.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.620+0000 7f3b38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:42.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.620+0000 7f3b38ff9640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:04:42.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.620+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b3c138250 con 0x7f3b3c0b1730 2024-03-25T18:04:42.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.632+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 969073160 0 0) 0x7f3b3c1387a0 con 0x7f3b3c14be90 2024-03-25T18:04:42.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.632+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b3c1383e0 con 0x7f3b3c14be90 2024-03-25T18:04:42.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.632+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3b34001b50 con 0x7f3b3c14be90 2024-03-25T18:04:42.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.632+0000 7f3b38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:42.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.632+0000 7f3b38ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:04:42.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.632+0000 7f3b38ff9640 10 monclient(hunting): dump: 2024-03-25T18:04:42.632 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:42.633 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:42.633 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:42.633 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:42.633 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:42.633 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:42.633 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:42.633 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:42.633 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:42.633 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:42.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.632+0000 7f3b38ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:04:42.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.634+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3894582826 0 0) 0x7f3b28004eb0 con 0x7f3b3c0b4d60 2024-03-25T18:04:42.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.634+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b3c1387a0 con 0x7f3b3c0b4d60 2024-03-25T18:04:42.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.634+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3b30001bc0 con 0x7f3b3c0b4d60 2024-03-25T18:04:42.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.634+0000 7f3b38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:42.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.635+0000 7f3b38ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:42.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.635+0000 7f3b38ff9640 10 monclient(hunting): dump: 2024-03-25T18:04:42.635 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:42.635 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:42.635 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:42.635 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:42.636 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:42.636 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:42.636 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:42.636 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:42.636 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:42.636 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:42.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.636+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1342578081 0 0) 0x7f3b3c138250 con 0x7f3b3c0b1730 2024-03-25T18:04:42.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.636+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b28004eb0 con 0x7f3b3c0b1730 2024-03-25T18:04:42.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.636+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3b2c0018f0 con 0x7f3b3c0b1730 2024-03-25T18:04:42.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.636+0000 7f3b38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:42.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.636+0000 7f3b38ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:42.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.636+0000 7f3b38ff9640 10 monclient(hunting): dump: 2024-03-25T18:04:42.637 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:42.637 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:42.637 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:42.637 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:42.637 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:42.637 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:42.637 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:42.638 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:42.638 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:42.638 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:42.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2586582049 0 0) 0x7f3b3c1383e0 con 0x7f3b3c14be90 2024-03-25T18:04:42.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 >> v1:172.21.15.18:6790/0 conn(0x7f3b3c0b4d60 legacy=0x7f3b3c1db260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:42.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 >> v1:172.21.15.169:6789/0 conn(0x7f3b3c0b1730 legacy=0x7f3b3c137b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:42.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b38ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:04:42.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b38ff9640 1 monclient: found mon.a 2024-03-25T18:04:42.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b38ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:42.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b38ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:04:42.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3c138930 con 0x7f3b3c14be90 2024-03-25T18:04:42.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b38ff9640 10 monclient: _finish_auth 0 2024-03-25T18:04:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b38ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:04:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b423b5640 5 monclient: authenticate success, global_id 4605 2024-03-25T18:04:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b423b5640 10 monclient: _renew_subs 2024-03-25T18:04:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b423b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:04:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b423b5640 1 -- 172.21.15.18:0/1996374310 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b3c143b60 con 0x7f3b3c14be90 2024-03-25T18:04:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b423b5640 10 client.4605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:04:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b423b5640 10 monclient: _renew_subs 2024-03-25T18:04:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b423b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:04:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b423b5640 1 -- 172.21.15.18:0/1996374310 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3c1dbb50 con 0x7f3b3c14be90 2024-03-25T18:04:42.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.647+0000 7f3b423b5640 1 librados: init done 2024-03-25T18:04:42.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.648+0000 7f3b423b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:42.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.648+0000 7f3b423b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:42.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.648+0000 7f3b423b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:04:42.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.648+0000 7f3b423b5640 1 -- 172.21.15.18:0/1996374310 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3c1db0b0 con 0x7f3b3c14be90 2024-03-25T18:04:42.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.982+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b34001fe0 con 0x7f3b3c14be90 2024-03-25T18:04:42.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.982+0000 7f3b38ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:42.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.982+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3b34003b30 con 0x7f3b3c14be90 2024-03-25T18:04:42.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.982+0000 7f3b38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:42.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.982+0000 7f3b38ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:04:42.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.982+0000 7f3b38ff9640 10 monclient: dump: 2024-03-25T18:04:42.983 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:42.983 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:42.983 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:42.984 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:42.984 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:42.984 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:42.984 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:42.984 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:42.984 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:42.984 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:42.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.982+0000 7f3b3a7fc640 4 set_mon_vals no callback set 2024-03-25T18:04:42.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.983+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f3b34019b00 con 0x7f3b3c14be90 2024-03-25T18:04:42.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.984+0000 7f3b3bfff640 10 monclient: get_auth_request con 0x7f3b2806e0b0 auth_method 0 2024-03-25T18:04:42.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.984+0000 7f3b38ff9640 10 client.4605.objecter ms_handle_connect 0x7f3b2806e0b0 2024-03-25T18:04:42.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.991+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f3b34053760 con 0x7f3b3c14be90 2024-03-25T18:04:42.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.992+0000 7f3b38ff9640 10 client.4605.objecter ms_dispatch 0x7f3b3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:04:42.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.992+0000 7f3b38ff9640 3 client.4605.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:04:42.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.992+0000 7f3b38ff9640 3 client.4605.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:04:42.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.992+0000 7f3b38ff9640 20 client.4605.objecter dump_active .. 0 homeless 2024-03-25T18:04:42.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.992+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3b3404f0a0 con 0x7f3b3c14be90 2024-03-25T18:04:42.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.992+0000 7f3b38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:42.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:42.992+0000 7f3b38ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:04:43.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.220+0000 7f3b423b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:43.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.220+0000 7f3b423b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:43.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.220+0000 7f3b423b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:04:43.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.220+0000 7f3b423b5640 1 -- 172.21.15.18:0/1996374310 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b3c1daea0 con 0x7f3b3c14be90 2024-03-25T18:04:43.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.264+0000 7f3b38ff9640 1 -- 172.21.15.18:0/1996374310 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f3b340511a0 con 0x7f3b3c14be90 2024-03-25T18:04:43.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.264+0000 7f3b38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:43.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.264+0000 7f3b38ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:04:43.264 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:04:43.265 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:04:43.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.266+0000 7f3b423b5640 10 librados: watch_flush enter 2024-03-25T18:04:43.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.266+0000 7f3b423b5640 10 librados: watch_flush exit 2024-03-25T18:04:43.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.266+0000 7f3b423b5640 20 client.4605.objecter shutdown clearing up homeless session... 2024-03-25T18:04:43.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.266+0000 7f3b423b5640 10 client.4605.objecter successfully canceled tick 2024-03-25T18:04:43.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.266+0000 7f3b423b5640 1 -- 172.21.15.18:0/1996374310 >> v1:172.21.15.18:6800/3721244451 conn(0x7f3b2806e0b0 legacy=0x7f3b28070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:43.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.266+0000 7f3b423b5640 10 monclient: shutdown 2024-03-25T18:04:43.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.266+0000 7f3b423b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:43.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.266+0000 7f3b423b5640 1 -- 172.21.15.18:0/1996374310 >> v1:172.21.15.18:6789/0 conn(0x7f3b3c14be90 legacy=0x7f3b3c130e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:43.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.266+0000 7f3b423b5640 1 -- 172.21.15.18:0/1996374310 shutdown_connections 2024-03-25T18:04:43.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.266+0000 7f3b423b5640 1 -- 172.21.15.18:0/1996374310 >> 172.21.15.18:0/1996374310 conn(0x7f3b3c0617a0 msgr2=0x7f3b3c0b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:43.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.267+0000 7f3b423b5640 1 -- 172.21.15.18:0/1996374310 shutdown_connections 2024-03-25T18:04:43.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.267+0000 7f3b423b5640 1 -- 172.21.15.18:0/1996374310 wait complete. 2024-03-25T18:04:43.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:43.267+0000 7f3b423b5640 1 librados: shutdown 2024-03-25T18:04:43.279 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:04:48.628 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:46.420 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:48.641 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:46.434 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:48.672 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:08:46.464 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:48.696 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:46.444 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:48.699 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:46.447 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:48.700 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:46.492 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:48.714 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:46.506 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:48.732 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:08:46.481 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:48.743 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:08:46.535 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:50.086 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:04:50.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.203+0000 7f9ba6301640 10 monclient: get_monmap_and_config 2024-03-25T18:04:50.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.203+0000 7f9ba6301640 10 monclient: build_initial_monmap 2024-03-25T18:04:50.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.203+0000 7f9ba6301640 10 monclient: monmap: 2024-03-25T18:04:50.203 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:04:50.204 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:50.204 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:04:50.203719+0000 2024-03-25T18:04:50.204 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:04:50.203719+0000 2024-03-25T18:04:50.204 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:50.204 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:50.204 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:04:50.204 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:04:50.204 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:04:50.204 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:50.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.204+0000 7f9ba6301640 1 Processor -- start 2024-03-25T18:04:50.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.204+0000 7f9ba6301640 1 -- start start 2024-03-25T18:04:50.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.204+0000 7f9ba6301640 10 monclient: init 2024-03-25T18:04:50.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.204+0000 7f9ba6301640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:50.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.204+0000 7f9ba6301640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:04:50.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.204+0000 7f9ba6301640 10 monclient(hunting): picked mon.noname-c con 0x7f9ba00b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T18:04:50.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.204+0000 7f9ba6301640 10 monclient(hunting): picked mon.noname-a con 0x7f9ba00b1730 addr v1:172.21.15.18:6789/0 2024-03-25T18:04:50.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.205+0000 7f9ba6301640 10 monclient(hunting): picked mon.noname-b con 0x7f9ba014be90 addr v1:172.21.15.18:6790/0 2024-03-25T18:04:50.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.205+0000 7f9ba6301640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba00af940 con 0x7f9ba00b1730 2024-03-25T18:04:50.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.205+0000 7f9ba6301640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba00b0a70 con 0x7f9ba00b4d60 2024-03-25T18:04:50.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.205+0000 7f9ba6301640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba00b0d50 con 0x7f9ba014be90 2024-03-25T18:04:50.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.205+0000 7f9ba6301640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:50.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.205+0000 7f9ba6301640 10 monclient(hunting): authenticate will time out at 7418.782827s 2024-03-25T18:04:50.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.209+0000 7f9b9ffff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f9ba00b4d60 0x7f9ba013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:50696/0 (socket says 172.21.15.18:50696) 2024-03-25T18:04:50.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.209+0000 7f9b9ffff640 1 -- 172.21.15.18:0/773116993 learned_addr learned my addr 172.21.15.18:0/773116993 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:04:50.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.224+0000 7f9b9effd640 1 -- 172.21.15.18:0/773116993 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793078751 0 0) 0x7f9ba00b0a70 con 0x7f9ba00b4d60 2024-03-25T18:04:50.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.224+0000 7f9b9effd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:50.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.224+0000 7f9b9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:50.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.224+0000 7f9b9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:50.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.224+0000 7f9b9effd640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:04:50.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.224+0000 7f9b9effd640 1 -- 172.21.15.18:0/773116993 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b88003520 con 0x7f9ba00b4d60 2024-03-25T18:04:50.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.226+0000 7f9b9effd640 1 -- 172.21.15.18:0/773116993 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696804005 0 0) 0x7f9ba00af940 con 0x7f9ba00b1730 2024-03-25T18:04:50.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.226+0000 7f9b9effd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:50.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.226+0000 7f9b9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:50.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.226+0000 7f9b9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:50.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.226+0000 7f9b9effd640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:04:50.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.226+0000 7f9b9effd640 1 -- 172.21.15.18:0/773116993 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ba00b0a70 con 0x7f9ba00b1730 2024-03-25T18:04:50.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.234+0000 7f9b9effd640 1 -- 172.21.15.18:0/773116993 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311365013 0 0) 0x7f9ba00b0d50 con 0x7f9ba014be90 2024-03-25T18:04:50.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.234+0000 7f9b9effd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:50.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.234+0000 7f9b9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:50.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.234+0000 7f9b9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:50.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.234+0000 7f9b9effd640 10 monclient(hunting): my global_id is 4610 2024-03-25T18:04:50.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.234+0000 7f9b9effd640 1 -- 172.21.15.18:0/773116993 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ba00af940 con 0x7f9ba014be90 2024-03-25T18:04:50.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9b9effd640 1 -- 172.21.15.18:0/773116993 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3846515500 0 0) 0x7f9b88003520 con 0x7f9ba00b4d60 2024-03-25T18:04:50.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9b9effd640 1 -- 172.21.15.18:0/773116993 >> v1:172.21.15.18:6790/0 conn(0x7f9ba014be90 legacy=0x7f9ba014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:50.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9b9effd640 1 -- 172.21.15.18:0/773116993 >> v1:172.21.15.18:6789/0 conn(0x7f9ba00b1730 legacy=0x7f9ba00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:50.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9b9effd640 10 monclient: _finish_hunting 0 2024-03-25T18:04:50.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9b9effd640 1 monclient: found mon.noname-c 2024-03-25T18:04:50.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9b9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:50.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9b9effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:04:50.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9b9effd640 1 -- 172.21.15.18:0/773116993 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba00b1030 con 0x7f9ba00b4d60 2024-03-25T18:04:50.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9b9effd640 10 monclient: _finish_auth 0 2024-03-25T18:04:50.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9b9effd640 10 monclient: _check_auth_tickets 2024-03-25T18:04:50.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9b9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:50.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9b9effd640 1 -- 172.21.15.18:0/773116993 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9b90001690 con 0x7f9ba00b4d60 2024-03-25T18:04:50.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9ba6301640 5 monclient: authenticate success, global_id 4594 2024-03-25T18:04:50.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9ba6301640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:50.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9b9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:50.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9b9effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:04:50.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9b9effd640 10 monclient: dump: 2024-03-25T18:04:50.241 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:50.241 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:50.241 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:50.241 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:50.241 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:50.241 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:50.241 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:50.241 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:50.241 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:50.241 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:50.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.239+0000 7f9ba6301640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:50.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.248+0000 7f9b9effd640 1 -- 172.21.15.18:0/773116993 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b900017f0 con 0x7f9ba00b4d60 2024-03-25T18:04:50.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.248+0000 7f9b9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:50.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.248+0000 7f9b9effd640 1 -- 172.21.15.18:0/773116993 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9b90001c00 con 0x7f9ba00b4d60 2024-03-25T18:04:50.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.248+0000 7f9ba6301640 10 monclient: get_monmap_and_config success 2024-03-25T18:04:50.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.248+0000 7f9ba6301640 4 set_mon_vals no callback set 2024-03-25T18:04:50.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.248+0000 7f9b9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:50.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.248+0000 7f9b9effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:50.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.248+0000 7f9b9effd640 10 monclient: dump: 2024-03-25T18:04:50.249 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:50.249 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:50.249 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:50.249 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:50.249 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:50.249 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:50.249 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:50.249 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:50.249 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:50.249 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:50.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.248+0000 7f9ba6301640 10 monclient: shutdown 2024-03-25T18:04:50.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.248+0000 7f9ba6301640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:50.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.248+0000 7f9ba6301640 1 -- 172.21.15.18:0/773116993 >> v1:172.21.15.169:6789/0 conn(0x7f9ba00b4d60 legacy=0x7f9ba013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:50.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.248+0000 7f9ba6301640 1 -- 172.21.15.18:0/773116993 shutdown_connections 2024-03-25T18:04:50.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.248+0000 7f9ba6301640 1 -- 172.21.15.18:0/773116993 >> 172.21.15.18:0/773116993 conn(0x7f9ba00617a0 msgr2=0x7f9ba0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:50.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.248+0000 7f9ba6301640 1 -- 172.21.15.18:0/773116993 shutdown_connections 2024-03-25T18:04:50.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.249+0000 7f9ba6301640 1 -- 172.21.15.18:0/773116993 wait complete. 2024-03-25T18:04:50.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.249+0000 7f9ba6301640 10 monclient: build_initial_monmap 2024-03-25T18:04:50.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.249+0000 7f9ba6301640 10 monclient: monmap: 2024-03-25T18:04:50.252 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:04:50.252 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:04:50.252 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:04:50.252 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:04:50.252 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:50.253 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:04:50.253 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:04:50.253 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:04:50.253 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:04:50.253 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:50.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.250+0000 7f9ba6301640 1 librados: starting msgr at 2024-03-25T18:04:50.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.250+0000 7f9ba6301640 1 librados: starting objecter 2024-03-25T18:04:50.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.250+0000 7f9ba6301640 1 Processor -- start 2024-03-25T18:04:50.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.250+0000 7f9ba6301640 1 -- start start 2024-03-25T18:04:50.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.250+0000 7f9ba6301640 1 librados: setting wanted keys 2024-03-25T18:04:50.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.250+0000 7f9ba6301640 1 librados: calling monclient init 2024-03-25T18:04:50.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.250+0000 7f9ba6301640 10 monclient: init 2024-03-25T18:04:50.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.251+0000 7f9ba6301640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:50.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.251+0000 7f9ba6301640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:04:50.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.251+0000 7f9ba6301640 10 monclient(hunting): picked mon.noname-a con 0x7f9ba00b1730 addr v1:172.21.15.18:6789/0 2024-03-25T18:04:50.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.251+0000 7f9ba6301640 10 monclient(hunting): picked mon.noname-b con 0x7f9ba00b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T18:04:50.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.251+0000 7f9ba6301640 10 monclient(hunting): picked mon.noname-c con 0x7f9ba014be90 addr v1:172.21.15.18:6790/0 2024-03-25T18:04:50.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.251+0000 7f9ba6301640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba0144150 con 0x7f9ba00b1730 2024-03-25T18:04:50.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.251+0000 7f9ba6301640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba0145840 con 0x7f9ba00b4d60 2024-03-25T18:04:50.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.251+0000 7f9ba6301640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba0143c30 con 0x7f9ba014be90 2024-03-25T18:04:50.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.251+0000 7f9ba6301640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:50.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.251+0000 7f9ba6301640 10 monclient(hunting): authenticate will time out at 7418.828939s 2024-03-25T18:04:50.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.257+0000 7f9b9ffff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f9ba00b1730 0x7f9ba012fef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:58720/0 (socket says 172.21.15.18:58720) 2024-03-25T18:04:50.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.257+0000 7f9b9ffff640 1 -- 172.21.15.18:0/169671845 learned_addr learned my addr 172.21.15.18:0/169671845 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:04:50.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.258+0000 7f9b9cff9640 10 client.?.objecter ms_handle_connect 0x7f9ba00b1730 2024-03-25T18:04:50.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.258+0000 7f9b9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:50.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.264+0000 7f9b9cff9640 10 client.?.objecter ms_handle_connect 0x7f9ba00b4d60 2024-03-25T18:04:50.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.264+0000 7f9b9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:50.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.270+0000 7f9b9cff9640 10 client.?.objecter ms_handle_connect 0x7f9ba014be90 2024-03-25T18:04:50.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.270+0000 7f9b9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:50.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.277+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449842766 0 0) 0x7f9ba0145840 con 0x7f9ba00b4d60 2024-03-25T18:04:50.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.277+0000 7f9b9cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:04:50.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.277+0000 7f9b9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:50.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.277+0000 7f9b9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:50.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.277+0000 7f9b9cff9640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:04:50.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.277+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b8c004eb0 con 0x7f9ba00b4d60 2024-03-25T18:04:50.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.278+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180667380 0 0) 0x7f9ba0144150 con 0x7f9ba00b1730 2024-03-25T18:04:50.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.278+0000 7f9b9cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:04:50.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.278+0000 7f9b9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:50.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.278+0000 7f9b9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:50.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.278+0000 7f9b9cff9640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:04:50.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.278+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ba0145840 con 0x7f9ba00b1730 2024-03-25T18:04:50.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.288+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450197765 0 0) 0x7f9ba0143c30 con 0x7f9ba014be90 2024-03-25T18:04:50.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.288+0000 7f9b9cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:04:50.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.288+0000 7f9b9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:50.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.288+0000 7f9b9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:50.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.288+0000 7f9b9cff9640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:04:50.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.288+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ba0144150 con 0x7f9ba014be90 2024-03-25T18:04:50.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.290+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3697387128 0 0) 0x7f9b8c004eb0 con 0x7f9ba00b4d60 2024-03-25T18:04:50.290 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.290+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ba0143c30 con 0x7f9ba00b4d60 2024-03-25T18:04:50.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.290+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9b94001970 con 0x7f9ba00b4d60 2024-03-25T18:04:50.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.290+0000 7f9b9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:50.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.290+0000 7f9b9cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:04:50.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.290+0000 7f9b9cff9640 10 monclient(hunting): dump: 2024-03-25T18:04:50.291 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:50.291 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:50.291 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:50.291 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:50.291 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:50.291 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:50.291 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:50.291 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:50.291 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:50.291 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:50.291 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.290+0000 7f9b9cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:04:50.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.296+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2314199686 0 0) 0x7f9ba0145840 con 0x7f9ba00b1730 2024-03-25T18:04:50.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.296+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b8c004eb0 con 0x7f9ba00b1730 2024-03-25T18:04:50.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.296+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9b90001b40 con 0x7f9ba00b1730 2024-03-25T18:04:50.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.296+0000 7f9b9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:50.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.296+0000 7f9b9cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:04:50.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.296+0000 7f9b9cff9640 10 monclient(hunting): dump: 2024-03-25T18:04:50.297 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:50.297 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:50.297 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:50.297 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:50.297 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:50.297 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:50.298 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:50.298 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:50.298 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:50.298 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:50.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 793644142 0 0) 0x7f9ba0143c30 con 0x7f9ba00b4d60 2024-03-25T18:04:50.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 >> v1:172.21.15.18:6790/0 conn(0x7f9ba014be90 legacy=0x7f9ba01e2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:50.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 >> v1:172.21.15.18:6789/0 conn(0x7f9ba00b1730 legacy=0x7f9ba012fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:50.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9b9cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:04:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9b9cff9640 1 monclient: found mon.b 2024-03-25T18:04:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9b9cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9b9cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba0143900 con 0x7f9ba00b4d60 2024-03-25T18:04:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9b9cff9640 10 monclient: _finish_auth 0 2024-03-25T18:04:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9b9cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:04:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9b9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9ba6301640 5 monclient: authenticate success, global_id 4597 2024-03-25T18:04:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9ba6301640 10 monclient: _renew_subs 2024-03-25T18:04:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9ba6301640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9ba6301640 1 -- 172.21.15.18:0/169671845 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba0137990 con 0x7f9ba00b4d60 2024-03-25T18:04:50.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9ba6301640 10 client.4597.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:04:50.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9ba6301640 10 monclient: _renew_subs 2024-03-25T18:04:50.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9ba6301640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:50.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9ba6301640 1 -- 172.21.15.18:0/169671845 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba01307e0 con 0x7f9ba00b4d60 2024-03-25T18:04:50.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.315+0000 7f9ba6301640 1 librados: init done 2024-03-25T18:04:50.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.317+0000 7f9ba6301640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:50.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.317+0000 7f9ba6301640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:50.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.317+0000 7f9ba6301640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:50.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.317+0000 7f9ba6301640 1 -- 172.21.15.18:0/169671845 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba01401f0 con 0x7f9ba00b4d60 2024-03-25T18:04:50.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.595+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b940035f0 con 0x7f9ba00b4d60 2024-03-25T18:04:50.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.595+0000 7f9b9cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:50.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.595+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9b94003900 con 0x7f9ba00b4d60 2024-03-25T18:04:50.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.595+0000 7f9b9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:50.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.595+0000 7f9b9cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:50.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.595+0000 7f9b9cff9640 10 monclient: dump: 2024-03-25T18:04:50.608 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:50.608 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:50.608 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:50.608 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:50.608 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:50.608 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:50.608 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:50.609 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:50.609 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:50.609 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:50.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.595+0000 7f9b9e7fc640 4 set_mon_vals no callback set 2024-03-25T18:04:50.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.596+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f9b940198d0 con 0x7f9ba00b4d60 2024-03-25T18:04:50.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.597+0000 7f9b9ffff640 10 monclient: get_auth_request con 0x7f9b8c06ded0 auth_method 0 2024-03-25T18:04:50.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.597+0000 7f9b9cff9640 10 client.4597.objecter ms_handle_connect 0x7f9b8c06ded0 2024-03-25T18:04:50.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.601+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f9b94053640 con 0x7f9ba00b4d60 2024-03-25T18:04:50.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.601+0000 7f9b9cff9640 10 client.4597.objecter ms_dispatch 0x7f9ba0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:04:50.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.601+0000 7f9b9cff9640 3 client.4597.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:04:50.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.601+0000 7f9b9cff9640 3 client.4597.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:04:50.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.601+0000 7f9b9cff9640 20 client.4597.objecter dump_active .. 0 homeless 2024-03-25T18:04:50.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.601+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b9404f150 con 0x7f9ba00b4d60 2024-03-25T18:04:50.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.601+0000 7f9b9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:50.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.601+0000 7f9b9cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:04:50.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.857+0000 7f9ba6301640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:50.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.857+0000 7f9ba6301640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:50.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.857+0000 7f9ba6301640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:50.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.857+0000 7f9ba6301640 1 -- 172.21.15.18:0/169671845 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ba0137b20 con 0x7f9ba00b4d60 2024-03-25T18:04:50.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.882+0000 7f9b9cff9640 1 -- 172.21.15.18:0/169671845 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f9b94051080 con 0x7f9ba00b4d60 2024-03-25T18:04:50.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.882+0000 7f9b9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:50.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.882+0000 7f9b9cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:04:50.883 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:04:50.883 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:04:50.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.884+0000 7f9ba6301640 10 librados: watch_flush enter 2024-03-25T18:04:50.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.884+0000 7f9ba6301640 10 librados: watch_flush exit 2024-03-25T18:04:50.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.884+0000 7f9ba6301640 20 client.4597.objecter shutdown clearing up homeless session... 2024-03-25T18:04:50.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.884+0000 7f9ba6301640 10 client.4597.objecter successfully canceled tick 2024-03-25T18:04:50.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.884+0000 7f9ba6301640 1 -- 172.21.15.18:0/169671845 >> v1:172.21.15.18:6800/3721244451 conn(0x7f9b8c06ded0 legacy=0x7f9b8c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:50.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.884+0000 7f9ba6301640 10 monclient: shutdown 2024-03-25T18:04:50.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.884+0000 7f9ba6301640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:50.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.884+0000 7f9ba6301640 1 -- 172.21.15.18:0/169671845 >> v1:172.21.15.169:6789/0 conn(0x7f9ba00b4d60 legacy=0x7f9ba01d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:50.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.884+0000 7f9ba6301640 1 -- 172.21.15.18:0/169671845 shutdown_connections 2024-03-25T18:04:50.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.884+0000 7f9ba6301640 1 -- 172.21.15.18:0/169671845 >> 172.21.15.18:0/169671845 conn(0x7f9ba00617a0 msgr2=0x7f9ba013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:50.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.884+0000 7f9ba6301640 1 -- 172.21.15.18:0/169671845 shutdown_connections 2024-03-25T18:04:50.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.884+0000 7f9ba6301640 1 -- 172.21.15.18:0/169671845 wait complete. 2024-03-25T18:04:50.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:50.884+0000 7f9ba6301640 1 librados: shutdown 2024-03-25T18:04:50.897 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:04:57.703 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:04:57.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.818+0000 7f7cdf425640 10 monclient: get_monmap_and_config 2024-03-25T18:04:57.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.818+0000 7f7cdf425640 10 monclient: build_initial_monmap 2024-03-25T18:04:57.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.819+0000 7f7cdf425640 10 monclient: monmap: 2024-03-25T18:04:57.821 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:04:57.821 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:57.821 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:04:57.820369+0000 2024-03-25T18:04:57.821 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:04:57.820369+0000 2024-03-25T18:04:57.821 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:57.821 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:57.821 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:04:57.821 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:04:57.821 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:04:57.821 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:57.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.819+0000 7f7cdf425640 1 Processor -- start 2024-03-25T18:04:57.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.820+0000 7f7cdf425640 1 -- start start 2024-03-25T18:04:57.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.820+0000 7f7cdf425640 10 monclient: init 2024-03-25T18:04:57.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.820+0000 7f7cdf425640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:57.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.820+0000 7f7cdf425640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:04:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.820+0000 7f7cdf425640 10 monclient(hunting): picked mon.noname-a con 0x7f7cd80a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:04:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.820+0000 7f7cdf425640 10 monclient(hunting): picked mon.noname-c con 0x7f7cd813f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:04:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.820+0000 7f7cdf425640 10 monclient(hunting): picked mon.noname-b con 0x7f7cd81430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:04:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.820+0000 7f7cdf425640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd8142520 con 0x7f7cd80a98f0 2024-03-25T18:04:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.821+0000 7f7cdf425640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd80b28a0 con 0x7f7cd813f710 2024-03-25T18:04:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.821+0000 7f7cdf425640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd80b2a30 con 0x7f7cd81430c0 2024-03-25T18:04:57.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.821+0000 7f7cdf425640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:57.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.821+0000 7f7cdf425640 10 monclient(hunting): authenticate will time out at 7426.399686s 2024-03-25T18:04:57.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.834+0000 7f7cdc9c0640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f7cd813f710 0x7f7cd81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:47074/0 (socket says 172.21.15.18:47074) 2024-03-25T18:04:57.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.834+0000 7f7cdc9c0640 1 -- 172.21.15.18:0/168910658 learned_addr learned my addr 172.21.15.18:0/168910658 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:04:57.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.840+0000 7f7ccffff640 1 -- 172.21.15.18:0/168910658 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771707652 0 0) 0x7f7cd80b2a30 con 0x7f7cd81430c0 2024-03-25T18:04:57.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.840+0000 7f7ccffff640 10 monclient(hunting): _negotiate 2024-03-25T18:04:57.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.840+0000 7f7ccffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:57.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.840+0000 7f7ccffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:57.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.840+0000 7f7ccffff640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:04:57.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.840+0000 7f7ccffff640 1 -- 172.21.15.18:0/168910658 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cc4003520 con 0x7f7cd81430c0 2024-03-25T18:04:57.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.846+0000 7f7ccffff640 1 -- 172.21.15.18:0/168910658 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219662170 0 0) 0x7f7cd80b28a0 con 0x7f7cd813f710 2024-03-25T18:04:57.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.846+0000 7f7ccffff640 10 monclient(hunting): _negotiate 2024-03-25T18:04:57.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.846+0000 7f7ccffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:57.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.846+0000 7f7ccffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:57.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.846+0000 7f7ccffff640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:04:57.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.846+0000 7f7ccffff640 1 -- 172.21.15.18:0/168910658 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd80b2a30 con 0x7f7cd813f710 2024-03-25T18:04:57.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.854+0000 7f7ccffff640 1 -- 172.21.15.18:0/168910658 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3400366006 0 0) 0x7f7cc4003520 con 0x7f7cd81430c0 2024-03-25T18:04:57.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.854+0000 7f7ccffff640 1 -- 172.21.15.18:0/168910658 >> v1:172.21.15.169:6789/0 conn(0x7f7cd813f710 legacy=0x7f7cd81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:57.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.854+0000 7f7ccffff640 1 -- 172.21.15.18:0/168910658 >> v1:172.21.15.18:6789/0 conn(0x7f7cd80a98f0 legacy=0x7f7cd80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:57.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.854+0000 7f7ccffff640 10 monclient: _finish_hunting 0 2024-03-25T18:04:57.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.854+0000 7f7ccffff640 1 monclient: found mon.noname-b 2024-03-25T18:04:57.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.854+0000 7f7ccffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:57.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.854+0000 7f7ccffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:04:57.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.854+0000 7f7ccffff640 1 -- 172.21.15.18:0/168910658 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd80b44c0 con 0x7f7cd81430c0 2024-03-25T18:04:57.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.854+0000 7f7ccffff640 10 monclient: _finish_auth 0 2024-03-25T18:04:57.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.854+0000 7f7ccffff640 10 monclient: _check_auth_tickets 2024-03-25T18:04:57.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.854+0000 7f7ccffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:57.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.854+0000 7f7ccffff640 1 -- 172.21.15.18:0/168910658 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f7cd0001a10 con 0x7f7cd81430c0 2024-03-25T18:04:57.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.854+0000 7f7ccffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:57.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.854+0000 7f7ccffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:04:57.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.854+0000 7f7ccffff640 10 monclient: dump: 2024-03-25T18:04:57.857 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:57.857 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:57.857 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:57.857 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:57.857 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:57.857 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:57.857 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:57.857 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:57.857 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:57.857 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:57.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.854+0000 7f7cdf425640 5 monclient: authenticate success, global_id 4616 2024-03-25T18:04:57.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.854+0000 7f7cdf425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:57.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.861+0000 7f7ccffff640 1 -- 172.21.15.18:0/168910658 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cd0001b70 con 0x7f7cd81430c0 2024-03-25T18:04:57.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.861+0000 7f7ccffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:57.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.861+0000 7f7cdf425640 10 monclient: get_monmap_and_config success 2024-03-25T18:04:57.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.861+0000 7f7cdf425640 4 set_mon_vals no callback set 2024-03-25T18:04:57.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.862+0000 7f7cdf425640 10 monclient: shutdown 2024-03-25T18:04:57.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.862+0000 7f7cdf425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:57.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.862+0000 7f7cdf425640 1 -- 172.21.15.18:0/168910658 >> v1:172.21.15.18:6790/0 conn(0x7f7cd81430c0 legacy=0x7f7cd814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:57.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.862+0000 7f7cdf425640 1 -- 172.21.15.18:0/168910658 shutdown_connections 2024-03-25T18:04:57.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.862+0000 7f7cdf425640 1 -- 172.21.15.18:0/168910658 >> 172.21.15.18:0/168910658 conn(0x7f7cd80b7ea0 msgr2=0x7f7cd80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:57.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.862+0000 7f7cdf425640 1 -- 172.21.15.18:0/168910658 shutdown_connections 2024-03-25T18:04:57.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.862+0000 7f7cdf425640 1 -- 172.21.15.18:0/168910658 wait complete. 2024-03-25T18:04:57.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.863+0000 7f7cdf425640 10 monclient: build_initial_monmap 2024-03-25T18:04:57.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.863+0000 7f7cdf425640 10 monclient: monmap: 2024-03-25T18:04:57.865 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:04:57.865 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:04:57.865 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:04:57.865 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:04:57.865 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:57.865 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:04:57.866 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:04:57.866 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:04:57.866 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:04:57.866 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:57.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.863+0000 7f7cdf425640 1 librados: starting msgr at 2024-03-25T18:04:57.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.863+0000 7f7cdf425640 1 librados: starting objecter 2024-03-25T18:04:57.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.864+0000 7f7cdf425640 1 Processor -- start 2024-03-25T18:04:57.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.864+0000 7f7cdf425640 1 -- start start 2024-03-25T18:04:57.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.864+0000 7f7cdf425640 1 librados: setting wanted keys 2024-03-25T18:04:57.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.864+0000 7f7cdf425640 1 librados: calling monclient init 2024-03-25T18:04:57.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.864+0000 7f7cdf425640 10 monclient: init 2024-03-25T18:04:57.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.864+0000 7f7cdf425640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:57.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.864+0000 7f7cdf425640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:04:57.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.864+0000 7f7cdf425640 10 monclient(hunting): picked mon.noname-b con 0x7f7cd80a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:04:57.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.865+0000 7f7cdf425640 10 monclient(hunting): picked mon.noname-c con 0x7f7cd813f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:04:57.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.865+0000 7f7cdf425640 10 monclient(hunting): picked mon.noname-a con 0x7f7cd81430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:04:57.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.865+0000 7f7cdf425640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd806f650 con 0x7f7cd81430c0 2024-03-25T18:04:57.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.865+0000 7f7cdf425640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd806f930 con 0x7f7cd80a98f0 2024-03-25T18:04:57.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.865+0000 7f7cdf425640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd806fc10 con 0x7f7cd813f710 2024-03-25T18:04:57.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.865+0000 7f7cdf425640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:57.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.865+0000 7f7cdf425640 10 monclient(hunting): authenticate will time out at 7426.443779s 2024-03-25T18:04:57.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.873+0000 7f7cdd9c2640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f7cd81430c0 0x7f7cd814d000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:52984/0 (socket says 172.21.15.18:52984) 2024-03-25T18:04:57.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.873+0000 7f7cdd9c2640 1 -- 172.21.15.18:0/2968334491 learned_addr learned my addr 172.21.15.18:0/2968334491 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:04:57.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.874+0000 7f7ccdffb640 10 client.?.objecter ms_handle_connect 0x7f7cd81430c0 2024-03-25T18:04:57.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.874+0000 7f7ccdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:57.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.881+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074678951 0 0) 0x7f7cd806f650 con 0x7f7cd81430c0 2024-03-25T18:04:57.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.881+0000 7f7ccdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:04:57.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.881+0000 7f7ccdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:57.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.881+0000 7f7ccdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:57.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.881+0000 7f7ccdffb640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:04:57.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.881+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cbc004eb0 con 0x7f7cd81430c0 2024-03-25T18:04:57.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.883+0000 7f7ccdffb640 10 client.?.objecter ms_handle_connect 0x7f7cd80a98f0 2024-03-25T18:04:57.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.883+0000 7f7ccdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:57.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.887+0000 7f7ccdffb640 10 client.?.objecter ms_handle_connect 0x7f7cd813f710 2024-03-25T18:04:57.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.887+0000 7f7ccdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:57.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.894+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 768238396 0 0) 0x7f7cbc004eb0 con 0x7f7cd81430c0 2024-03-25T18:04:57.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.894+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cd806f650 con 0x7f7cd81430c0 2024-03-25T18:04:57.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.894+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f7cd0001f80 con 0x7f7cd81430c0 2024-03-25T18:04:57.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.894+0000 7f7ccdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:57.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.894+0000 7f7ccdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:04:57.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.894+0000 7f7ccdffb640 10 monclient(hunting): dump: 2024-03-25T18:04:57.896 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:57.896 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:57.896 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:57.896 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:57.896 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:57.896 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:57.896 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:57.896 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:57.896 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:57.896 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:57.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.894+0000 7f7ccdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:04:57.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.895+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502236375 0 0) 0x7f7cd806f930 con 0x7f7cd80a98f0 2024-03-25T18:04:57.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.895+0000 7f7ccdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:04:57.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.895+0000 7f7ccdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:57.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.895+0000 7f7ccdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:57.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.895+0000 7f7ccdffb640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:04:57.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.895+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cbc004eb0 con 0x7f7cd80a98f0 2024-03-25T18:04:57.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.905+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159158710 0 0) 0x7f7cd806fc10 con 0x7f7cd813f710 2024-03-25T18:04:57.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.905+0000 7f7ccdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:04:57.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.905+0000 7f7ccdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:57.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.905+0000 7f7ccdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:57.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.905+0000 7f7ccdffb640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:04:57.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.905+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd806f930 con 0x7f7cd813f710 2024-03-25T18:04:57.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.912+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2524980954 0 0) 0x7f7cbc004eb0 con 0x7f7cd80a98f0 2024-03-25T18:04:57.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.912+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cd806fc10 con 0x7f7cd80a98f0 2024-03-25T18:04:57.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.913+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f7cd4001870 con 0x7f7cd80a98f0 2024-03-25T18:04:57.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.913+0000 7f7ccdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:57.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.913+0000 7f7ccdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:57.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.913+0000 7f7ccdffb640 10 monclient(hunting): dump: 2024-03-25T18:04:57.914 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:57.914 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:57.914 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:57.915 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:57.915 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:57.915 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:57.915 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:57.915 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:57.915 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:57.915 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:57.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.926+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1299491451 0 0) 0x7f7cd806fc10 con 0x7f7cd80a98f0 2024-03-25T18:04:57.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.926+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 >> v1:172.21.15.18:6790/0 conn(0x7f7cd813f710 legacy=0x7f7cd814c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:57.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.926+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 >> v1:172.21.15.18:6789/0 conn(0x7f7cd81430c0 legacy=0x7f7cd814d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:57.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.926+0000 7f7ccdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:04:57.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.926+0000 7f7ccdffb640 1 monclient: found mon.b 2024-03-25T18:04:57.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.926+0000 7f7ccdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:57.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.926+0000 7f7ccdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:57.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.926+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd814c3d0 con 0x7f7cd80a98f0 2024-03-25T18:04:57.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.926+0000 7f7ccdffb640 10 monclient: _finish_auth 0 2024-03-25T18:04:57.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.926+0000 7f7ccdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:04:57.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.926+0000 7f7ccdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:57.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.926+0000 7f7cdf425640 5 monclient: authenticate success, global_id 4603 2024-03-25T18:04:57.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.926+0000 7f7cdf425640 10 monclient: _renew_subs 2024-03-25T18:04:57.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.926+0000 7f7cdf425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:57.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.926+0000 7f7cdf425640 1 -- 172.21.15.18:0/2968334491 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd806dea0 con 0x7f7cd80a98f0 2024-03-25T18:04:57.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.927+0000 7f7cdf425640 10 client.4603.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:04:57.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.927+0000 7f7cdf425640 10 monclient: _renew_subs 2024-03-25T18:04:57.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.927+0000 7f7cdf425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:57.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.927+0000 7f7cdf425640 1 -- 172.21.15.18:0/2968334491 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd806e1a0 con 0x7f7cd80a98f0 2024-03-25T18:04:57.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.927+0000 7f7cdf425640 1 librados: init done 2024-03-25T18:04:57.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.928+0000 7f7cdf425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:57.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.928+0000 7f7cdf425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:57.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.928+0000 7f7cdf425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:57.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:57.928+0000 7f7cdf425640 1 -- 172.21.15.18:0/2968334491 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd80627b0 con 0x7f7cd80a98f0 2024-03-25T18:04:58.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.210+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cd40035f0 con 0x7f7cd80a98f0 2024-03-25T18:04:58.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.210+0000 7f7ccdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:58.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.210+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f7cd4001d80 con 0x7f7cd80a98f0 2024-03-25T18:04:58.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.210+0000 7f7ccdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:58.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.210+0000 7f7ccdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:58.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.210+0000 7f7ccf7fe640 4 set_mon_vals no callback set 2024-03-25T18:04:58.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.210+0000 7f7ccdffb640 10 monclient: dump: 2024-03-25T18:04:58.212 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:04:58.212 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:04:58.212 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:58.212 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:04:58.212 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:58.212 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:04:58.212 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:04:58.212 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:04:58.212 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:04:58.212 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:04:58.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.210+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f7cd4019770 con 0x7f7cd80a98f0 2024-03-25T18:04:58.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.211+0000 7f7cdc9c0640 10 monclient: get_auth_request con 0x7f7cbc06df30 auth_method 0 2024-03-25T18:04:58.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.211+0000 7f7ccdffb640 10 client.4603.objecter ms_handle_connect 0x7f7cbc06df30 2024-03-25T18:04:58.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.216+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f7cd4053580 con 0x7f7cd80a98f0 2024-03-25T18:04:58.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.216+0000 7f7ccdffb640 10 client.4603.objecter ms_dispatch 0x7f7cd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:04:58.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.216+0000 7f7ccdffb640 3 client.4603.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:04:58.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.216+0000 7f7ccdffb640 3 client.4603.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:04:58.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.216+0000 7f7ccdffb640 20 client.4603.objecter dump_active .. 0 homeless 2024-03-25T18:04:58.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.216+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7cd404eec0 con 0x7f7cd80a98f0 2024-03-25T18:04:58.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.216+0000 7f7ccdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:58.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.216+0000 7f7ccdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:04:58.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.472+0000 7f7cdf425640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:58.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.472+0000 7f7cdf425640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:58.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.472+0000 7f7cdf425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:04:58.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.472+0000 7f7cdf425640 1 -- 172.21.15.18:0/2968334491 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cd813e2e0 con 0x7f7cd80a98f0 2024-03-25T18:04:58.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.498+0000 7f7ccdffb640 1 -- 172.21.15.18:0/2968334491 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f7cd4050fc0 con 0x7f7cd80a98f0 2024-03-25T18:04:58.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.498+0000 7f7ccdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:58.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.498+0000 7f7ccdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:04:58.500 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:04:58.500 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:04:58.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.500+0000 7f7cdf425640 10 librados: watch_flush enter 2024-03-25T18:04:58.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.500+0000 7f7cdf425640 10 librados: watch_flush exit 2024-03-25T18:04:58.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.500+0000 7f7cdf425640 20 client.4603.objecter shutdown clearing up homeless session... 2024-03-25T18:04:58.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.500+0000 7f7cdf425640 10 client.4603.objecter successfully canceled tick 2024-03-25T18:04:58.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.500+0000 7f7cdf425640 1 -- 172.21.15.18:0/2968334491 >> v1:172.21.15.18:6800/3721244451 conn(0x7f7cbc06df30 legacy=0x7f7cbc0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:58.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.501+0000 7f7cdf425640 10 monclient: shutdown 2024-03-25T18:04:58.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.501+0000 7f7cdf425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:58.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.501+0000 7f7cdf425640 1 -- 172.21.15.18:0/2968334491 >> v1:172.21.15.169:6789/0 conn(0x7f7cd80a98f0 legacy=0x7f7cd814e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:58.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.501+0000 7f7cdf425640 1 -- 172.21.15.18:0/2968334491 shutdown_connections 2024-03-25T18:04:58.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.501+0000 7f7cdf425640 1 -- 172.21.15.18:0/2968334491 >> 172.21.15.18:0/2968334491 conn(0x7f7cd80b7ea0 msgr2=0x7f7cd8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:58.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.501+0000 7f7cdf425640 1 -- 172.21.15.18:0/2968334491 shutdown_connections 2024-03-25T18:04:58.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.501+0000 7f7cdf425640 1 -- 172.21.15.18:0/2968334491 wait complete. 2024-03-25T18:04:58.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:04:58.501+0000 7f7cdf425640 1 librados: shutdown 2024-03-25T18:04:58.515 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:05.323 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:05.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.437+0000 7f9fb06fb640 10 monclient: get_monmap_and_config 2024-03-25T18:05:05.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.437+0000 7f9fb06fb640 10 monclient: build_initial_monmap 2024-03-25T18:05:05.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.437+0000 7f9fb06fb640 10 monclient: monmap: 2024-03-25T18:05:05.439 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:05:05.439 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:05.439 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:05:05.439222+0000 2024-03-25T18:05:05.439 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:05:05.439222+0000 2024-03-25T18:05:05.439 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:05.439 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:05.439 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:05:05.439 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:05:05.440 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:05:05.440 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:05.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.438+0000 7f9fb06fb640 1 Processor -- start 2024-03-25T18:05:05.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.438+0000 7f9fb06fb640 1 -- start start 2024-03-25T18:05:05.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.438+0000 7f9fb06fb640 10 monclient: init 2024-03-25T18:05:05.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.439+0000 7f9fb06fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:05.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.439+0000 7f9fb06fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:05:05.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.439+0000 7f9fb06fb640 10 monclient(hunting): picked mon.noname-a con 0x7f9fa80b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T18:05:05.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.439+0000 7f9fb06fb640 10 monclient(hunting): picked mon.noname-b con 0x7f9fa80b1730 addr v1:172.21.15.18:6790/0 2024-03-25T18:05:05.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.439+0000 7f9fb06fb640 10 monclient(hunting): picked mon.noname-c con 0x7f9fa814be90 addr v1:172.21.15.169:6789/0 2024-03-25T18:05:05.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.439+0000 7f9fb06fb640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa80af940 con 0x7f9fa80b4d60 2024-03-25T18:05:05.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.439+0000 7f9fb06fb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa80b0a70 con 0x7f9fa814be90 2024-03-25T18:05:05.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.439+0000 7f9fb06fb640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa80b0d50 con 0x7f9fa80b1730 2024-03-25T18:05:05.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.439+0000 7f9fb06fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:05.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.439+0000 7f9fb06fb640 10 monclient(hunting): authenticate will time out at 7434.018554s 2024-03-25T18:05:05.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.450+0000 7f9fadc96640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f9fa80b1730 0x7f9fa80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:53320/0 (socket says 172.21.15.18:53320) 2024-03-25T18:05:05.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.450+0000 7f9fadc96640 1 -- 172.21.15.18:0/3593532084 learned_addr learned my addr 172.21.15.18:0/3593532084 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:05:05.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.463+0000 7f9fad495640 1 -- 172.21.15.18:0/3593532084 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350706166 0 0) 0x7f9fa80b0d50 con 0x7f9fa80b1730 2024-03-25T18:05:05.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.463+0000 7f9fad495640 10 monclient(hunting): _negotiate 2024-03-25T18:05:05.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.463+0000 7f9fad495640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:05.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.463+0000 7f9fad495640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:05.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.463+0000 7f9fad495640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:05:05.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.463+0000 7f9fad495640 1 -- 172.21.15.18:0/3593532084 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f94003520 con 0x7f9fa80b1730 2024-03-25T18:05:05.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.475+0000 7f9fad495640 1 -- 172.21.15.18:0/3593532084 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077098002 0 0) 0x7f9fa80af940 con 0x7f9fa80b4d60 2024-03-25T18:05:05.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.475+0000 7f9fad495640 10 monclient(hunting): _negotiate 2024-03-25T18:05:05.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.475+0000 7f9fad495640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:05.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.475+0000 7f9fad495640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:05.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.475+0000 7f9fad495640 10 monclient(hunting): my global_id is 4620 2024-03-25T18:05:05.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.475+0000 7f9fad495640 1 -- 172.21.15.18:0/3593532084 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa80b0d50 con 0x7f9fa80b4d60 2024-03-25T18:05:05.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.477+0000 7f9fad495640 1 -- 172.21.15.18:0/3593532084 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977857066 0 0) 0x7f9fa80b0a70 con 0x7f9fa814be90 2024-03-25T18:05:05.478 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.477+0000 7f9fad495640 10 monclient(hunting): _negotiate 2024-03-25T18:05:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.477+0000 7f9fad495640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.477+0000 7f9fad495640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.477+0000 7f9fad495640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:05:05.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.477+0000 7f9fad495640 1 -- 172.21.15.18:0/3593532084 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa80af940 con 0x7f9fa814be90 2024-03-25T18:05:05.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.487+0000 7f9fad495640 1 -- 172.21.15.18:0/3593532084 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 425660851 0 0) 0x7f9fa80b0d50 con 0x7f9fa80b4d60 2024-03-25T18:05:05.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.487+0000 7f9fad495640 1 -- 172.21.15.18:0/3593532084 >> v1:172.21.15.18:6790/0 conn(0x7f9fa80b1730 legacy=0x7f9fa80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:05.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.487+0000 7f9fad495640 1 -- 172.21.15.18:0/3593532084 >> v1:172.21.15.169:6789/0 conn(0x7f9fa814be90 legacy=0x7f9fa814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:05.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.487+0000 7f9fad495640 10 monclient: _finish_hunting 0 2024-03-25T18:05:05.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.487+0000 7f9fad495640 1 monclient: found mon.noname-a 2024-03-25T18:05:05.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.487+0000 7f9fad495640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:05.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.487+0000 7f9fad495640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:05:05.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.487+0000 7f9fad495640 1 -- 172.21.15.18:0/3593532084 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa80b1030 con 0x7f9fa80b4d60 2024-03-25T18:05:05.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.487+0000 7f9fad495640 10 monclient: _finish_auth 0 2024-03-25T18:05:05.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.487+0000 7f9fad495640 10 monclient: _check_auth_tickets 2024-03-25T18:05:05.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.487+0000 7f9fad495640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:05.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.487+0000 7f9fad495640 1 -- 172.21.15.18:0/3593532084 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9fa4001670 con 0x7f9fa80b4d60 2024-03-25T18:05:05.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.487+0000 7f9fad495640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:05.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.487+0000 7f9fad495640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:05.490 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.487+0000 7f9fad495640 10 monclient: dump: 2024-03-25T18:05:05.490 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:05.490 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:05.490 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:05.490 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:05.490 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:05.490 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:05.490 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:05.490 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:05.490 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:05.490 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:05.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.487+0000 7f9fb06fb640 5 monclient: authenticate success, global_id 4620 2024-03-25T18:05:05.491 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.487+0000 7f9fb06fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:05.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.516+0000 7f9fad495640 1 -- 172.21.15.18:0/3593532084 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fa40017d0 con 0x7f9fa80b4d60 2024-03-25T18:05:05.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.516+0000 7f9fad495640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:05.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.516+0000 7f9fad495640 1 -- 172.21.15.18:0/3593532084 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9fa4001be0 con 0x7f9fa80b4d60 2024-03-25T18:05:05.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.516+0000 7f9fad495640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:05.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.516+0000 7f9fad495640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:05.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.516+0000 7f9fad495640 10 monclient: dump: 2024-03-25T18:05:05.518 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:05.518 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:05.518 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:05.518 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:05.518 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:05.519 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:05.519 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:05.519 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:05.519 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:05.519 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:05.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.516+0000 7f9fb06fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:05.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.516+0000 7f9fb06fb640 4 set_mon_vals no callback set 2024-03-25T18:05:05.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.517+0000 7f9fb06fb640 10 monclient: shutdown 2024-03-25T18:05:05.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.517+0000 7f9fb06fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:05.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.517+0000 7f9fb06fb640 1 -- 172.21.15.18:0/3593532084 >> v1:172.21.15.18:6789/0 conn(0x7f9fa80b4d60 legacy=0x7f9fa813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:05.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.517+0000 7f9fb06fb640 1 -- 172.21.15.18:0/3593532084 shutdown_connections 2024-03-25T18:05:05.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.517+0000 7f9fb06fb640 1 -- 172.21.15.18:0/3593532084 >> 172.21.15.18:0/3593532084 conn(0x7f9fa80617a0 msgr2=0x7f9fa8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:05.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.517+0000 7f9fb06fb640 1 -- 172.21.15.18:0/3593532084 shutdown_connections 2024-03-25T18:05:05.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.517+0000 7f9fb06fb640 1 -- 172.21.15.18:0/3593532084 wait complete. 2024-03-25T18:05:05.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.518+0000 7f9fb06fb640 10 monclient: build_initial_monmap 2024-03-25T18:05:05.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.518+0000 7f9fb06fb640 10 monclient: monmap: 2024-03-25T18:05:05.522 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:05:05.522 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:05.522 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:05:05.522 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:05:05.522 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:05.522 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:05:05.522 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:05:05.522 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:05:05.522 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:05:05.522 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:05.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.518+0000 7f9fb06fb640 1 librados: starting msgr at 2024-03-25T18:05:05.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.518+0000 7f9fb06fb640 1 librados: starting objecter 2024-03-25T18:05:05.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.518+0000 7f9fb06fb640 1 Processor -- start 2024-03-25T18:05:05.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.519+0000 7f9fb06fb640 1 -- start start 2024-03-25T18:05:05.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.519+0000 7f9fb06fb640 1 librados: setting wanted keys 2024-03-25T18:05:05.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.519+0000 7f9fb06fb640 1 librados: calling monclient init 2024-03-25T18:05:05.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.519+0000 7f9fb06fb640 10 monclient: init 2024-03-25T18:05:05.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.519+0000 7f9fb06fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:05.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.519+0000 7f9fb06fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:05:05.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.519+0000 7f9fb06fb640 10 monclient(hunting): picked mon.noname-c con 0x7f9fa80b1730 addr v1:172.21.15.18:6790/0 2024-03-25T18:05:05.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.519+0000 7f9fb06fb640 10 monclient(hunting): picked mon.noname-b con 0x7f9fa80b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T18:05:05.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.519+0000 7f9fb06fb640 10 monclient(hunting): picked mon.noname-a con 0x7f9fa814be90 addr v1:172.21.15.18:6789/0 2024-03-25T18:05:05.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.519+0000 7f9fb06fb640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa8143d40 con 0x7f9fa814be90 2024-03-25T18:05:05.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.519+0000 7f9fb06fb640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa8140f00 con 0x7f9fa80b4d60 2024-03-25T18:05:05.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.519+0000 7f9fb06fb640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa8138460 con 0x7f9fa80b1730 2024-03-25T18:05:05.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.519+0000 7f9fb06fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:05.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.519+0000 7f9fb06fb640 10 monclient(hunting): authenticate will time out at 7434.098430s 2024-03-25T18:05:05.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.537+0000 7f9fae497640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f9fa80b1730 0x7f9fa8137c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:53390/0 (socket says 172.21.15.18:53390) 2024-03-25T18:05:05.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.537+0000 7f9fae497640 1 -- 172.21.15.18:0/1750047519 learned_addr learned my addr 172.21.15.18:0/1750047519 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:05:05.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.540+0000 7f9f9effd640 10 client.?.objecter ms_handle_connect 0x7f9fa80b1730 2024-03-25T18:05:05.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.540+0000 7f9f9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:05.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.544+0000 7f9f9effd640 10 client.?.objecter ms_handle_connect 0x7f9fa80b4d60 2024-03-25T18:05:05.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.544+0000 7f9f9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:05.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.544+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192705407 0 0) 0x7f9fa8138460 con 0x7f9fa80b1730 2024-03-25T18:05:05.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.544+0000 7f9f9effd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:05.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.544+0000 7f9f9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:05.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.544+0000 7f9f9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:05.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.544+0000 7f9f9effd640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:05:05.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.544+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f8c004eb0 con 0x7f9fa80b1730 2024-03-25T18:05:05.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.549+0000 7f9f9effd640 10 client.?.objecter ms_handle_connect 0x7f9fa814be90 2024-03-25T18:05:05.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.549+0000 7f9f9effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:05.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.557+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929867530 0 0) 0x7f9fa8140f00 con 0x7f9fa80b4d60 2024-03-25T18:05:05.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.557+0000 7f9f9effd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:05.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.557+0000 7f9f9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:05.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.557+0000 7f9f9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:05.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.557+0000 7f9f9effd640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:05:05.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.557+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa8138460 con 0x7f9fa80b4d60 2024-03-25T18:05:05.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.559+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745308730 0 0) 0x7f9fa8143d40 con 0x7f9fa814be90 2024-03-25T18:05:05.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.559+0000 7f9f9effd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:05.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.559+0000 7f9f9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:05.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.559+0000 7f9f9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:05.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.559+0000 7f9f9effd640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:05:05.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.559+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa8140f00 con 0x7f9fa814be90 2024-03-25T18:05:05.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.564+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 723421903 0 0) 0x7f9f8c004eb0 con 0x7f9fa80b1730 2024-03-25T18:05:05.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.564+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fa8143d40 con 0x7f9fa80b1730 2024-03-25T18:05:05.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.565+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9fa4001670 con 0x7f9fa80b1730 2024-03-25T18:05:05.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.565+0000 7f9f9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:05.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.565+0000 7f9f9effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:05.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.565+0000 7f9f9effd640 10 monclient(hunting): dump: 2024-03-25T18:05:05.566 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:05.566 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:05.566 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:05.567 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:05.567 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:05.567 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:05.567 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:05.567 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:05.567 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:05.567 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:05.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.565+0000 7f9f9effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:05.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.572+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1787705153 0 0) 0x7f9fa8143d40 con 0x7f9fa80b1730 2024-03-25T18:05:05.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.572+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 >> v1:172.21.15.169:6789/0 conn(0x7f9fa80b4d60 legacy=0x7f9fa8142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:05.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.572+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 >> v1:172.21.15.18:6789/0 conn(0x7f9fa814be90 legacy=0x7f9fa81d7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:05.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.572+0000 7f9f9effd640 10 monclient: _finish_hunting 0 2024-03-25T18:05:05.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.572+0000 7f9f9effd640 1 monclient: found mon.c 2024-03-25T18:05:05.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.572+0000 7f9f9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:05.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.572+0000 7f9f9effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:05:05.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.572+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa8141130 con 0x7f9fa80b1730 2024-03-25T18:05:05.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.572+0000 7f9f9effd640 10 monclient: _finish_auth 0 2024-03-25T18:05:05.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.572+0000 7f9f9effd640 10 monclient: _check_auth_tickets 2024-03-25T18:05:05.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.572+0000 7f9f9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:05.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.572+0000 7f9fb06fb640 5 monclient: authenticate success, global_id 4625 2024-03-25T18:05:05.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.572+0000 7f9fb06fb640 10 monclient: _renew_subs 2024-03-25T18:05:05.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.572+0000 7f9fb06fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:05:05.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.572+0000 7f9fb06fb640 1 -- 172.21.15.18:0/1750047519 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa8137660 con 0x7f9fa80b1730 2024-03-25T18:05:05.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.573+0000 7f9fb06fb640 10 client.4625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:05.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.573+0000 7f9fb06fb640 10 monclient: _renew_subs 2024-03-25T18:05:05.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.573+0000 7f9fb06fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:05:05.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.573+0000 7f9fb06fb640 1 -- 172.21.15.18:0/1750047519 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa8143030 con 0x7f9fa80b1730 2024-03-25T18:05:05.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.573+0000 7f9fb06fb640 1 librados: init done 2024-03-25T18:05:05.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.573+0000 7f9fb06fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:05.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.573+0000 7f9fb06fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:05.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.573+0000 7f9fb06fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:05:05.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.573+0000 7f9fb06fb640 1 -- 172.21.15.18:0/1750047519 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa8143330 con 0x7f9fa80b1730 2024-03-25T18:05:05.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.872+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fa40038a0 con 0x7f9fa80b1730 2024-03-25T18:05:05.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.872+0000 7f9f9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:05.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.872+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f9fa4003c70 con 0x7f9fa80b1730 2024-03-25T18:05:05.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.872+0000 7f9f9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:05.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.872+0000 7f9f9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:05.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.872+0000 7f9f9effd640 10 monclient: dump: 2024-03-25T18:05:05.874 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:05.874 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:05.874 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:05.874 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:05.874 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:05.874 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:05.875 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:05.875 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:05.875 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:05.875 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:05.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.872+0000 7f9facc94640 4 set_mon_vals no callback set 2024-03-25T18:05:05.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.873+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f9fa4003e90 con 0x7f9fa80b1730 2024-03-25T18:05:05.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.874+0000 7f9fadc96640 10 monclient: get_auth_request con 0x7f9f8c06ddf0 auth_method 0 2024-03-25T18:05:05.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.874+0000 7f9f9effd640 10 client.4625.objecter ms_handle_connect 0x7f9f8c06ddf0 2024-03-25T18:05:05.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.879+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f9fa4054940 con 0x7f9fa80b1730 2024-03-25T18:05:05.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.879+0000 7f9f9effd640 10 client.4625.objecter ms_dispatch 0x7f9fa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:05.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.879+0000 7f9f9effd640 3 client.4625.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:05.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.879+0000 7f9f9effd640 3 client.4625.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:05.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.879+0000 7f9f9effd640 20 client.4625.objecter dump_active .. 0 homeless 2024-03-25T18:05:05.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.879+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9fa4050450 con 0x7f9fa80b1730 2024-03-25T18:05:05.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.879+0000 7f9f9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:05.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:05.879+0000 7f9f9effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:06.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.132+0000 7f9fb06fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:06.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.132+0000 7f9fb06fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:06.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.132+0000 7f9fb06fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:05:06.134 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.132+0000 7f9fb06fb640 1 -- 172.21.15.18:0/1750047519 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fa80627b0 con 0x7f9fa80b1730 2024-03-25T18:05:06.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.264+0000 7f9f9effd640 1 -- 172.21.15.18:0/1750047519 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f9fa4052380 con 0x7f9fa80b1730 2024-03-25T18:05:06.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.264+0000 7f9f9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:06.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.264+0000 7f9f9effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:06.266 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:05:06.266 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:06.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.266+0000 7f9fb06fb640 10 librados: watch_flush enter 2024-03-25T18:05:06.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.266+0000 7f9fb06fb640 10 librados: watch_flush exit 2024-03-25T18:05:06.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.266+0000 7f9fb06fb640 20 client.4625.objecter shutdown clearing up homeless session... 2024-03-25T18:05:06.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.266+0000 7f9fb06fb640 10 client.4625.objecter successfully canceled tick 2024-03-25T18:05:06.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.266+0000 7f9fb06fb640 1 -- 172.21.15.18:0/1750047519 >> v1:172.21.15.18:6800/3721244451 conn(0x7f9f8c06ddf0 legacy=0x7f9f8c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:06.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.266+0000 7f9fb06fb640 10 monclient: shutdown 2024-03-25T18:05:06.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.266+0000 7f9fb06fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:06.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.266+0000 7f9fb06fb640 1 -- 172.21.15.18:0/1750047519 >> v1:172.21.15.18:6790/0 conn(0x7f9fa80b1730 legacy=0x7f9fa8137c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:06.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.267+0000 7f9fb06fb640 1 -- 172.21.15.18:0/1750047519 shutdown_connections 2024-03-25T18:05:06.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.267+0000 7f9fb06fb640 1 -- 172.21.15.18:0/1750047519 >> 172.21.15.18:0/1750047519 conn(0x7f9fa80617a0 msgr2=0x7f9fa80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:06.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.267+0000 7f9fb06fb640 1 -- 172.21.15.18:0/1750047519 shutdown_connections 2024-03-25T18:05:06.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.267+0000 7f9fb06fb640 1 -- 172.21.15.18:0/1750047519 wait complete. 2024-03-25T18:05:06.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:06.267+0000 7f9fb06fb640 1 librados: shutdown 2024-03-25T18:05:06.279 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:08.645 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:06.437 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:08.694 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:06.486 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:08.721 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:06.513 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:08.745 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:06.537 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:08.756 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:06.504 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:08.757 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:06.549 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:08.773 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:06.521 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:08.793 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:06.586 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:08.811 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:06.559 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:13.086 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:13.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.198+0000 7f5f967e8640 10 monclient: get_monmap_and_config 2024-03-25T18:05:13.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.198+0000 7f5f967e8640 10 monclient: build_initial_monmap 2024-03-25T18:05:13.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.198+0000 7f5f967e8640 10 monclient: monmap: 2024-03-25T18:05:13.199 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:05:13.200 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:13.200 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:05:13.199402+0000 2024-03-25T18:05:13.200 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:05:13.199402+0000 2024-03-25T18:05:13.200 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:13.200 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:13.200 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:05:13.200 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:05:13.200 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:05:13.200 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:13.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.198+0000 7f5f967e8640 1 Processor -- start 2024-03-25T18:05:13.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.198+0000 7f5f967e8640 1 -- start start 2024-03-25T18:05:13.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.198+0000 7f5f967e8640 10 monclient: init 2024-03-25T18:05:13.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.199+0000 7f5f967e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:13.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.199+0000 7f5f967e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:05:13.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.199+0000 7f5f967e8640 10 monclient(hunting): picked mon.noname-c con 0x7f5f900b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T18:05:13.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.199+0000 7f5f967e8640 10 monclient(hunting): picked mon.noname-b con 0x7f5f900b1730 addr v1:172.21.15.18:6790/0 2024-03-25T18:05:13.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.199+0000 7f5f967e8640 10 monclient(hunting): picked mon.noname-a con 0x7f5f9014be90 addr v1:172.21.15.18:6789/0 2024-03-25T18:05:13.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.199+0000 7f5f967e8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f900af940 con 0x7f5f9014be90 2024-03-25T18:05:13.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.199+0000 7f5f967e8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f900b0a70 con 0x7f5f900b4d60 2024-03-25T18:05:13.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.199+0000 7f5f967e8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f900b0d50 con 0x7f5f900b1730 2024-03-25T18:05:13.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.199+0000 7f5f967e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:13.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.199+0000 7f5f967e8640 10 monclient(hunting): authenticate will time out at 7441.778202s 2024-03-25T18:05:13.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.200+0000 7f5f8f7fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f5f900b1730 0x7f5f900b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:57826/0 (socket says 172.21.15.18:57826) 2024-03-25T18:05:13.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.200+0000 7f5f8ffff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f5f900b4d60 0x7f5f9013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:60978/0 (socket says 172.21.15.18:60978) 2024-03-25T18:05:13.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.200+0000 7f5f8f7fe640 1 -- 172.21.15.18:0/2619740292 learned_addr learned my addr 172.21.15.18:0/2619740292 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:05:13.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.278+0000 7f5f8effd640 1 -- 172.21.15.18:0/2619740292 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693114124 0 0) 0x7f5f900af940 con 0x7f5f9014be90 2024-03-25T18:05:13.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.278+0000 7f5f8effd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:13.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.278+0000 7f5f8effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:13.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.278+0000 7f5f8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:13.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.278+0000 7f5f8effd640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:05:13.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.278+0000 7f5f8effd640 1 -- 172.21.15.18:0/2619740292 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f7c003520 con 0x7f5f9014be90 2024-03-25T18:05:13.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.301+0000 7f5f8effd640 1 -- 172.21.15.18:0/2619740292 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3670828615 0 0) 0x7f5f7c003520 con 0x7f5f9014be90 2024-03-25T18:05:13.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.301+0000 7f5f8effd640 1 -- 172.21.15.18:0/2619740292 >> v1:172.21.15.18:6790/0 conn(0x7f5f900b1730 legacy=0x7f5f900b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.301+0000 7f5f8effd640 1 -- 172.21.15.18:0/2619740292 >> v1:172.21.15.169:6789/0 conn(0x7f5f900b4d60 legacy=0x7f5f9013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.301+0000 7f5f8effd640 10 monclient: _finish_hunting 0 2024-03-25T18:05:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.301+0000 7f5f8effd640 1 monclient: found mon.noname-a 2024-03-25T18:05:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.301+0000 7f5f8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.301+0000 7f5f8effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:05:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.301+0000 7f5f8effd640 1 -- 172.21.15.18:0/2619740292 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f900b1030 con 0x7f5f9014be90 2024-03-25T18:05:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.301+0000 7f5f8effd640 10 monclient: _finish_auth 0 2024-03-25T18:05:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.301+0000 7f5f8effd640 10 monclient: _check_auth_tickets 2024-03-25T18:05:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.301+0000 7f5f8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.301+0000 7f5f8effd640 1 -- 172.21.15.18:0/2619740292 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5f88001a10 con 0x7f5f9014be90 2024-03-25T18:05:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.301+0000 7f5f8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.301+0000 7f5f8effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:13.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.301+0000 7f5f8effd640 10 monclient: dump: 2024-03-25T18:05:13.303 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:13.304 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:13.304 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:13.304 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:13.304 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:13.304 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:13.304 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:13.304 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:13.304 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:13.304 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.301+0000 7f5f967e8640 5 monclient: authenticate success, global_id 4626 2024-03-25T18:05:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.301+0000 7f5f967e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:13.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.319+0000 7f5f8effd640 1 -- 172.21.15.18:0/2619740292 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f88001b70 con 0x7f5f9014be90 2024-03-25T18:05:13.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.319+0000 7f5f8effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:13.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.319+0000 7f5f8effd640 1 -- 172.21.15.18:0/2619740292 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5f88001fc0 con 0x7f5f9014be90 2024-03-25T18:05:13.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.319+0000 7f5f8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:13.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.319+0000 7f5f8effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:13.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.319+0000 7f5f8effd640 10 monclient: dump: 2024-03-25T18:05:13.321 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:13.321 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:13.321 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:13.321 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:13.321 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:13.321 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:13.321 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:13.321 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:13.321 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:13.321 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:13.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.319+0000 7f5f967e8640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:13.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.319+0000 7f5f967e8640 4 set_mon_vals no callback set 2024-03-25T18:05:13.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.319+0000 7f5f967e8640 10 monclient: shutdown 2024-03-25T18:05:13.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.319+0000 7f5f967e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:13.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.319+0000 7f5f967e8640 1 -- 172.21.15.18:0/2619740292 >> v1:172.21.15.18:6789/0 conn(0x7f5f9014be90 legacy=0x7f5f9014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:13.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.320+0000 7f5f967e8640 1 -- 172.21.15.18:0/2619740292 shutdown_connections 2024-03-25T18:05:13.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.320+0000 7f5f967e8640 1 -- 172.21.15.18:0/2619740292 >> 172.21.15.18:0/2619740292 conn(0x7f5f900617a0 msgr2=0x7f5f90061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:13.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.320+0000 7f5f967e8640 1 -- 172.21.15.18:0/2619740292 shutdown_connections 2024-03-25T18:05:13.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.320+0000 7f5f967e8640 1 -- 172.21.15.18:0/2619740292 wait complete. 2024-03-25T18:05:13.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.321+0000 7f5f967e8640 10 monclient: build_initial_monmap 2024-03-25T18:05:13.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.321+0000 7f5f967e8640 10 monclient: monmap: 2024-03-25T18:05:13.323 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:05:13.323 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:13.323 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:05:13.324 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:05:13.324 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:13.324 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:05:13.324 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:05:13.324 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:05:13.324 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:05:13.324 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:13.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.321+0000 7f5f967e8640 1 librados: starting msgr at 2024-03-25T18:05:13.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.321+0000 7f5f967e8640 1 librados: starting objecter 2024-03-25T18:05:13.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.321+0000 7f5f967e8640 1 Processor -- start 2024-03-25T18:05:13.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.321+0000 7f5f967e8640 1 -- start start 2024-03-25T18:05:13.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.321+0000 7f5f967e8640 1 librados: setting wanted keys 2024-03-25T18:05:13.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.321+0000 7f5f967e8640 1 librados: calling monclient init 2024-03-25T18:05:13.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.321+0000 7f5f967e8640 10 monclient: init 2024-03-25T18:05:13.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.322+0000 7f5f967e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:13.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.322+0000 7f5f967e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:05:13.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.322+0000 7f5f967e8640 10 monclient(hunting): picked mon.noname-a con 0x7f5f900b1730 addr v1:172.21.15.18:6789/0 2024-03-25T18:05:13.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.322+0000 7f5f967e8640 10 monclient(hunting): picked mon.noname-c con 0x7f5f900b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T18:05:13.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.322+0000 7f5f967e8640 10 monclient(hunting): picked mon.noname-b con 0x7f5f9014be90 addr v1:172.21.15.169:6789/0 2024-03-25T18:05:13.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.322+0000 7f5f967e8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f901307c0 con 0x7f5f900b1730 2024-03-25T18:05:13.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.322+0000 7f5f967e8640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f90138f90 con 0x7f5f9014be90 2024-03-25T18:05:13.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.322+0000 7f5f967e8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f90132ab0 con 0x7f5f900b4d60 2024-03-25T18:05:13.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.322+0000 7f5f967e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:13.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.322+0000 7f5f967e8640 10 monclient(hunting): authenticate will time out at 7441.901208s 2024-03-25T18:05:13.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.333+0000 7f5f8f7fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f5f900b4d60 0x7f5f9012f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:57888/0 (socket says 172.21.15.18:57888) 2024-03-25T18:05:13.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.333+0000 7f5f8f7fe640 1 -- 172.21.15.18:0/150088911 learned_addr learned my addr 172.21.15.18:0/150088911 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:05:13.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.333+0000 7f5f8cff9640 10 client.?.objecter ms_handle_connect 0x7f5f900b4d60 2024-03-25T18:05:13.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.333+0000 7f5f8cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:13.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.335+0000 7f5f8cff9640 10 client.?.objecter ms_handle_connect 0x7f5f9014be90 2024-03-25T18:05:13.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.335+0000 7f5f8cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:13.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.339+0000 7f5f8cff9640 10 client.?.objecter ms_handle_connect 0x7f5f900b1730 2024-03-25T18:05:13.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.339+0000 7f5f8cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:13.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.407+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034140618 0 0) 0x7f5f901307c0 con 0x7f5f900b1730 2024-03-25T18:05:13.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.407+0000 7f5f8cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:05:13.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.407+0000 7f5f8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:13.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.407+0000 7f5f8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:13.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.407+0000 7f5f8cff9640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:05:13.409 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.407+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f70004eb0 con 0x7f5f900b1730 2024-03-25T18:05:13.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.413+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106427369 0 0) 0x7f5f90132ab0 con 0x7f5f900b4d60 2024-03-25T18:05:13.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.413+0000 7f5f8cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:05:13.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.413+0000 7f5f8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:13.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.413+0000 7f5f8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:13.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.413+0000 7f5f8cff9640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:05:13.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.413+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f901307c0 con 0x7f5f900b4d60 2024-03-25T18:05:13.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.431+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1438949478 0 0) 0x7f5f70004eb0 con 0x7f5f900b1730 2024-03-25T18:05:13.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.431+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f90132ab0 con 0x7f5f900b1730 2024-03-25T18:05:13.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.431+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5f80001860 con 0x7f5f900b1730 2024-03-25T18:05:13.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.431+0000 7f5f8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:13.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.431+0000 7f5f8cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:13.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.431+0000 7f5f8cff9640 10 monclient(hunting): dump: 2024-03-25T18:05:13.433 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:13.433 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:13.433 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:13.433 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:13.433 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:13.433 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:13.433 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:13.433 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:13.433 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:13.434 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:13.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.431+0000 7f5f8cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:13.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.434+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 65617916 0 0) 0x7f5f901307c0 con 0x7f5f900b4d60 2024-03-25T18:05:13.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.434+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f70004eb0 con 0x7f5f900b4d60 2024-03-25T18:05:13.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.434+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5f780018f0 con 0x7f5f900b4d60 2024-03-25T18:05:13.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.434+0000 7f5f8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:13.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.434+0000 7f5f8cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:13.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.434+0000 7f5f8cff9640 10 monclient(hunting): dump: 2024-03-25T18:05:13.436 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:13.436 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:13.436 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:13.436 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:13.436 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:13.436 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:13.436 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:13.436 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:13.436 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:13.436 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:13.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4113467399 0 0) 0x7f5f90132ab0 con 0x7f5f900b1730 2024-03-25T18:05:13.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 >> v1:172.21.15.18:6790/0 conn(0x7f5f900b4d60 legacy=0x7f5f9012f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:13.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 >> v1:172.21.15.169:6789/0 conn(0x7f5f9014be90 legacy=0x7f5f901300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:13.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f8cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:05:13.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f8cff9640 1 monclient: found mon.a 2024-03-25T18:05:13.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f8cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:13.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f8cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:13.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f90133a40 con 0x7f5f900b1730 2024-03-25T18:05:13.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f8cff9640 10 monclient: _finish_auth 0 2024-03-25T18:05:13.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f8cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:05:13.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:13.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f967e8640 5 monclient: authenticate success, global_id 4629 2024-03-25T18:05:13.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f967e8640 10 monclient: _renew_subs 2024-03-25T18:05:13.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f967e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:13.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f967e8640 1 -- 172.21.15.18:0/150088911 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f901316c0 con 0x7f5f900b1730 2024-03-25T18:05:13.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f967e8640 10 client.4629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:13.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f967e8640 10 monclient: _renew_subs 2024-03-25T18:05:13.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f967e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:13.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f967e8640 1 -- 172.21.15.18:0/150088911 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f90137860 con 0x7f5f900b1730 2024-03-25T18:05:13.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.442+0000 7f5f967e8640 1 librados: init done 2024-03-25T18:05:13.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.443+0000 7f5f967e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:13.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.443+0000 7f5f967e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:13.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.443+0000 7f5f967e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:13.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.443+0000 7f5f967e8640 1 -- 172.21.15.18:0/150088911 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f90132c40 con 0x7f5f900b1730 2024-03-25T18:05:13.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.448+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f80001d50 con 0x7f5f900b1730 2024-03-25T18:05:13.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.448+0000 7f5f8cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:13.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.448+0000 7f5f8e7fc640 4 set_mon_vals no callback set 2024-03-25T18:05:13.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.760+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5f80004980 con 0x7f5f900b1730 2024-03-25T18:05:13.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.760+0000 7f5f8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:13.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.760+0000 7f5f8cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:13.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.760+0000 7f5f8cff9640 10 monclient: dump: 2024-03-25T18:05:13.762 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:13.762 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:13.762 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:13.762 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:13.762 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:13.762 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:13.762 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:13.762 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:13.762 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:13.762 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:13.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.761+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f5f80004b60 con 0x7f5f900b1730 2024-03-25T18:05:13.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.761+0000 7f5f8f7fe640 10 monclient: get_auth_request con 0x7f5f7006ded0 auth_method 0 2024-03-25T18:05:13.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.762+0000 7f5f8cff9640 10 client.4629.objecter ms_handle_connect 0x7f5f7006ded0 2024-03-25T18:05:13.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.770+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f5f80054790 con 0x7f5f900b1730 2024-03-25T18:05:13.771 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.770+0000 7f5f8cff9640 10 client.4629.objecter ms_dispatch 0x7f5f90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:13.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.770+0000 7f5f8cff9640 3 client.4629.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:13.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.770+0000 7f5f8cff9640 3 client.4629.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:13.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.770+0000 7f5f8cff9640 20 client.4629.objecter dump_active .. 0 homeless 2024-03-25T18:05:13.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.770+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f800500d0 con 0x7f5f900b1730 2024-03-25T18:05:13.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.770+0000 7f5f8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:13.772 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:13.770+0000 7f5f8cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:14.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.029+0000 7f5f967e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:14.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.029+0000 7f5f967e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:14.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.029+0000 7f5f967e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:14.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.029+0000 7f5f967e8640 1 -- 172.21.15.18:0/150088911 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f900627b0 con 0x7f5f900b1730 2024-03-25T18:05:14.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.068+0000 7f5f8cff9640 1 -- 172.21.15.18:0/150088911 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f5f800521d0 con 0x7f5f900b1730 2024-03-25T18:05:14.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.068+0000 7f5f8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:14.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.068+0000 7f5f8cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:14.070 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:05:14.070 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:14.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.070+0000 7f5f967e8640 10 librados: watch_flush enter 2024-03-25T18:05:14.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.070+0000 7f5f967e8640 10 librados: watch_flush exit 2024-03-25T18:05:14.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.070+0000 7f5f967e8640 20 client.4629.objecter shutdown clearing up homeless session... 2024-03-25T18:05:14.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.070+0000 7f5f967e8640 10 client.4629.objecter successfully canceled tick 2024-03-25T18:05:14.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.070+0000 7f5f967e8640 1 -- 172.21.15.18:0/150088911 >> v1:172.21.15.18:6800/3721244451 conn(0x7f5f7006ded0 legacy=0x7f5f70070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:14.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.070+0000 7f5f967e8640 10 monclient: shutdown 2024-03-25T18:05:14.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.070+0000 7f5f967e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:14.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.070+0000 7f5f967e8640 1 -- 172.21.15.18:0/150088911 >> v1:172.21.15.18:6789/0 conn(0x7f5f900b1730 legacy=0x7f5f901393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:14.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.070+0000 7f5f967e8640 1 -- 172.21.15.18:0/150088911 shutdown_connections 2024-03-25T18:05:14.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.070+0000 7f5f967e8640 1 -- 172.21.15.18:0/150088911 >> 172.21.15.18:0/150088911 conn(0x7f5f900617a0 msgr2=0x7f5f9013c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:14.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.071+0000 7f5f967e8640 1 -- 172.21.15.18:0/150088911 shutdown_connections 2024-03-25T18:05:14.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.071+0000 7f5f967e8640 1 -- 172.21.15.18:0/150088911 wait complete. 2024-03-25T18:05:14.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:14.071+0000 7f5f967e8640 1 librados: shutdown 2024-03-25T18:05:14.084 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:20.891 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:21.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.006+0000 7fc0b12d9640 10 monclient: get_monmap_and_config 2024-03-25T18:05:21.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.006+0000 7fc0b12d9640 10 monclient: build_initial_monmap 2024-03-25T18:05:21.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.007+0000 7fc0b12d9640 10 monclient: monmap: 2024-03-25T18:05:21.008 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:05:21.009 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:21.009 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:05:21.008375+0000 2024-03-25T18:05:21.009 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:05:21.008375+0000 2024-03-25T18:05:21.009 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:21.009 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:21.009 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:05:21.009 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:05:21.009 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:05:21.009 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:21.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.007+0000 7fc0b12d9640 1 Processor -- start 2024-03-25T18:05:21.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.007+0000 7fc0b12d9640 1 -- start start 2024-03-25T18:05:21.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.007+0000 7fc0b12d9640 10 monclient: init 2024-03-25T18:05:21.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.008+0000 7fc0b12d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:21.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.008+0000 7fc0b12d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:05:21.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.008+0000 7fc0b12d9640 10 monclient(hunting): picked mon.noname-a con 0x7fc0ac0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:05:21.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.008+0000 7fc0b12d9640 10 monclient(hunting): picked mon.noname-c con 0x7fc0ac13f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:05:21.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.008+0000 7fc0b12d9640 10 monclient(hunting): picked mon.noname-b con 0x7fc0ac1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:05:21.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.008+0000 7fc0b12d9640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ac142520 con 0x7fc0ac0a98f0 2024-03-25T18:05:21.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.008+0000 7fc0b12d9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ac0b28a0 con 0x7fc0ac13f710 2024-03-25T18:05:21.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.008+0000 7fc0b12d9640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ac0b2a30 con 0x7fc0ac1430c0 2024-03-25T18:05:21.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.008+0000 7fc0b12d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:21.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.008+0000 7fc0b12d9640 10 monclient(hunting): authenticate will time out at 7449.587244s 2024-03-25T18:05:21.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.019+0000 7fc0aa7fc640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc0ac13f710 0x7fc0ac1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:37440/0 (socket says 172.21.15.18:37440) 2024-03-25T18:05:21.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.019+0000 7fc0aa7fc640 1 -- 172.21.15.18:0/347985480 learned_addr learned my addr 172.21.15.18:0/347985480 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:05:21.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.032+0000 7fc0a9ffb640 1 -- 172.21.15.18:0/347985480 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428655368 0 0) 0x7fc0ac0b28a0 con 0x7fc0ac13f710 2024-03-25T18:05:21.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.033+0000 7fc0a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:21.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.033+0000 7fc0a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:21.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.033+0000 7fc0a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:21.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.033+0000 7fc0a9ffb640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:05:21.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.033+0000 7fc0a9ffb640 1 -- 172.21.15.18:0/347985480 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc094003520 con 0x7fc0ac13f710 2024-03-25T18:05:21.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0a9ffb640 1 -- 172.21.15.18:0/347985480 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2279893374 0 0) 0x7fc094003520 con 0x7fc0ac13f710 2024-03-25T18:05:21.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0a9ffb640 1 -- 172.21.15.18:0/347985480 >> v1:172.21.15.18:6790/0 conn(0x7fc0ac1430c0 legacy=0x7fc0ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:21.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0a9ffb640 1 -- 172.21.15.18:0/347985480 >> v1:172.21.15.18:6789/0 conn(0x7fc0ac0a98f0 legacy=0x7fc0ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:21.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0a9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:05:21.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0a9ffb640 1 monclient: found mon.noname-c 2024-03-25T18:05:21.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:21.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0a9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:05:21.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0a9ffb640 1 -- 172.21.15.18:0/347985480 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ac0b44c0 con 0x7fc0ac13f710 2024-03-25T18:05:21.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0a9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:05:21.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0a9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:05:21.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:21.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0a9ffb640 1 -- 172.21.15.18:0/347985480 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc0a0001690 con 0x7fc0ac13f710 2024-03-25T18:05:21.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0b12d9640 5 monclient: authenticate success, global_id 4618 2024-03-25T18:05:21.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0b12d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:21.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:21.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0a9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:21.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0a9ffb640 10 monclient: dump: 2024-03-25T18:05:21.047 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:21.047 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:21.047 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:21.047 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:21.047 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:21.047 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:21.047 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:21.047 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:21.047 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:21.048 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:21.048 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.044+0000 7fc0b12d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:21.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.059+0000 7fc0a9ffb640 1 -- 172.21.15.18:0/347985480 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0a00017f0 con 0x7fc0ac13f710 2024-03-25T18:05:21.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.059+0000 7fc0a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:21.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.059+0000 7fc0a9ffb640 1 -- 172.21.15.18:0/347985480 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc0a0001c00 con 0x7fc0ac13f710 2024-03-25T18:05:21.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.059+0000 7fc0b12d9640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:21.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.059+0000 7fc0b12d9640 4 set_mon_vals no callback set 2024-03-25T18:05:21.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.059+0000 7fc0a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:21.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.059+0000 7fc0a9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:21.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.059+0000 7fc0a9ffb640 10 monclient: dump: 2024-03-25T18:05:21.062 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:21.062 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:21.062 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:21.062 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:21.062 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:21.062 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:21.062 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:21.062 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:21.062 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:21.062 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:21.063 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.060+0000 7fc0b12d9640 10 monclient: shutdown 2024-03-25T18:05:21.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.060+0000 7fc0b12d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:21.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.060+0000 7fc0b12d9640 1 -- 172.21.15.18:0/347985480 >> v1:172.21.15.169:6789/0 conn(0x7fc0ac13f710 legacy=0x7fc0ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:21.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.060+0000 7fc0b12d9640 1 -- 172.21.15.18:0/347985480 shutdown_connections 2024-03-25T18:05:21.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.060+0000 7fc0b12d9640 1 -- 172.21.15.18:0/347985480 >> 172.21.15.18:0/347985480 conn(0x7fc0ac0b7ea0 msgr2=0x7fc0ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:21.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.060+0000 7fc0b12d9640 1 -- 172.21.15.18:0/347985480 shutdown_connections 2024-03-25T18:05:21.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.060+0000 7fc0b12d9640 1 -- 172.21.15.18:0/347985480 wait complete. 2024-03-25T18:05:21.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.061+0000 7fc0b12d9640 10 monclient: build_initial_monmap 2024-03-25T18:05:21.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.061+0000 7fc0b12d9640 10 monclient: monmap: 2024-03-25T18:05:21.064 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:05:21.064 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:21.064 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:05:21.064 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:05:21.064 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:21.065 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:05:21.065 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:05:21.065 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:05:21.065 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:05:21.065 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:21.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.062+0000 7fc0b12d9640 1 librados: starting msgr at 2024-03-25T18:05:21.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.062+0000 7fc0b12d9640 1 librados: starting objecter 2024-03-25T18:05:21.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.062+0000 7fc0b12d9640 1 Processor -- start 2024-03-25T18:05:21.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.062+0000 7fc0b12d9640 1 -- start start 2024-03-25T18:05:21.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.062+0000 7fc0b12d9640 1 librados: setting wanted keys 2024-03-25T18:05:21.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.062+0000 7fc0b12d9640 1 librados: calling monclient init 2024-03-25T18:05:21.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.062+0000 7fc0b12d9640 10 monclient: init 2024-03-25T18:05:21.066 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.063+0000 7fc0b12d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:21.066 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.063+0000 7fc0b12d9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:05:21.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.063+0000 7fc0b12d9640 10 monclient(hunting): picked mon.noname-c con 0x7fc0ac0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:05:21.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.063+0000 7fc0b12d9640 10 monclient(hunting): picked mon.noname-b con 0x7fc0ac13f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:05:21.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.063+0000 7fc0b12d9640 10 monclient(hunting): picked mon.noname-a con 0x7fc0ac1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:05:21.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.063+0000 7fc0b12d9640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ac0b1870 con 0x7fc0ac1430c0 2024-03-25T18:05:21.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.063+0000 7fc0b12d9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ac0ac910 con 0x7fc0ac13f710 2024-03-25T18:05:21.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.063+0000 7fc0b12d9640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ac0abb00 con 0x7fc0ac0a98f0 2024-03-25T18:05:21.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.063+0000 7fc0b12d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:21.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.063+0000 7fc0b12d9640 10 monclient(hunting): authenticate will time out at 7449.642084s 2024-03-25T18:05:21.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.066+0000 7fc0aa7fc640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc0ac13f710 0x7fc0ac0ac200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:37460/0 (socket says 172.21.15.18:37460) 2024-03-25T18:05:21.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.066+0000 7fc0aa7fc640 1 -- 172.21.15.18:0/860096116 learned_addr learned my addr 172.21.15.18:0/860096116 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:05:21.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.067+0000 7fc0937fe640 10 client.?.objecter ms_handle_connect 0x7fc0ac13f710 2024-03-25T18:05:21.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.067+0000 7fc0937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:21.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.071+0000 7fc0937fe640 10 client.?.objecter ms_handle_connect 0x7fc0ac0a98f0 2024-03-25T18:05:21.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.071+0000 7fc0937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:21.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.080+0000 7fc0937fe640 10 client.?.objecter ms_handle_connect 0x7fc0ac1430c0 2024-03-25T18:05:21.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.080+0000 7fc0937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:21.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.144+0000 7fc0937fe640 1 -- 172.21.15.18:0/860096116 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351397081 0 0) 0x7fc0ac0b2a30 con 0x7fc0ac1430c0 2024-03-25T18:05:21.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.144+0000 7fc0937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:21.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.144+0000 7fc0937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:21.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.144+0000 7fc0937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:21.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.144+0000 7fc0937fe640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:05:21.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.144+0000 7fc0937fe640 1 -- 172.21.15.18:0/860096116 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc098004eb0 con 0x7fc0ac1430c0 2024-03-25T18:05:21.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.164+0000 7fc0937fe640 1 -- 172.21.15.18:0/860096116 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2366984383 0 0) 0x7fc098004eb0 con 0x7fc0ac1430c0 2024-03-25T18:05:21.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.164+0000 7fc0937fe640 1 -- 172.21.15.18:0/860096116 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0ac0b2a30 con 0x7fc0ac1430c0 2024-03-25T18:05:21.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.164+0000 7fc0937fe640 1 -- 172.21.15.18:0/860096116 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc0a4001980 con 0x7fc0ac1430c0 2024-03-25T18:05:21.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.164+0000 7fc0937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:21.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.164+0000 7fc0937fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:21.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.164+0000 7fc0937fe640 10 monclient(hunting): dump: 2024-03-25T18:05:21.166 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:21.166 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:21.166 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:21.166 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:21.166 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:21.166 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:21.166 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:21.166 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:21.166 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:21.167 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:21.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.164+0000 7fc0937fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:21.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0937fe640 1 -- 172.21.15.18:0/860096116 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 337352597 0 0) 0x7fc0ac0b2a30 con 0x7fc0ac1430c0 2024-03-25T18:05:21.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0937fe640 1 -- 172.21.15.18:0/860096116 >> v1:172.21.15.18:6790/0 conn(0x7fc0ac0a98f0 legacy=0x7fc0ac138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:21.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0937fe640 1 -- 172.21.15.18:0/860096116 >> v1:172.21.15.169:6789/0 conn(0x7fc0ac13f710 legacy=0x7fc0ac0ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:21.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0937fe640 10 monclient: _finish_hunting 0 2024-03-25T18:05:21.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0937fe640 1 monclient: found mon.a 2024-03-25T18:05:21.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:21.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0937fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:21.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0937fe640 1 -- 172.21.15.18:0/860096116 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ac0abde0 con 0x7fc0ac1430c0 2024-03-25T18:05:21.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0937fe640 10 monclient: _finish_auth 0 2024-03-25T18:05:21.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0937fe640 10 monclient: _check_auth_tickets 2024-03-25T18:05:21.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:21.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0b12d9640 5 monclient: authenticate success, global_id 4635 2024-03-25T18:05:21.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0b12d9640 10 monclient: _renew_subs 2024-03-25T18:05:21.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0b12d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:21.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0b12d9640 1 -- 172.21.15.18:0/860096116 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0ac138f40 con 0x7fc0ac1430c0 2024-03-25T18:05:21.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0b12d9640 10 client.4635.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:21.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0b12d9640 10 monclient: _renew_subs 2024-03-25T18:05:21.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0b12d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:21.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0b12d9640 1 -- 172.21.15.18:0/860096116 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0ac1363f0 con 0x7fc0ac1430c0 2024-03-25T18:05:21.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.187+0000 7fc0b12d9640 1 librados: init done 2024-03-25T18:05:21.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.188+0000 7fc0b12d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:21.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.188+0000 7fc0b12d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:21.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.188+0000 7fc0b12d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:21.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.188+0000 7fc0b12d9640 1 -- 172.21.15.18:0/860096116 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0ac0627b0 con 0x7fc0ac1430c0 2024-03-25T18:05:21.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.515+0000 7fc0937fe640 1 -- 172.21.15.18:0/860096116 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0a4001ed0 con 0x7fc0ac1430c0 2024-03-25T18:05:21.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.515+0000 7fc0937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:21.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.515+0000 7fc0937fe640 1 -- 172.21.15.18:0/860096116 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc0a4003b10 con 0x7fc0ac1430c0 2024-03-25T18:05:21.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.515+0000 7fc0937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:21.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.515+0000 7fc0937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:21.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.515+0000 7fc0937fe640 10 monclient: dump: 2024-03-25T18:05:21.517 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:21.517 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:21.518 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:21.518 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:21.518 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:21.518 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:21.518 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:21.518 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:21.518 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:21.518 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:21.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.515+0000 7fc0a97fa640 4 set_mon_vals no callback set 2024-03-25T18:05:21.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.516+0000 7fc0937fe640 1 -- 172.21.15.18:0/860096116 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fc0a4003d30 con 0x7fc0ac1430c0 2024-03-25T18:05:21.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.517+0000 7fc0aaffd640 10 monclient: get_auth_request con 0x7fc09806dd90 auth_method 0 2024-03-25T18:05:21.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.517+0000 7fc0937fe640 10 client.4635.objecter ms_handle_connect 0x7fc09806dd90 2024-03-25T18:05:21.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.522+0000 7fc0937fe640 1 -- 172.21.15.18:0/860096116 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fc0a4053660 con 0x7fc0ac1430c0 2024-03-25T18:05:21.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.522+0000 7fc0937fe640 10 client.4635.objecter ms_dispatch 0x7fc0ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:21.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.522+0000 7fc0937fe640 3 client.4635.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:21.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.522+0000 7fc0937fe640 3 client.4635.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:21.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.522+0000 7fc0937fe640 20 client.4635.objecter dump_active .. 0 homeless 2024-03-25T18:05:21.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.522+0000 7fc0937fe640 1 -- 172.21.15.18:0/860096116 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc0a404f170 con 0x7fc0ac1430c0 2024-03-25T18:05:21.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.522+0000 7fc0937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:21.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.522+0000 7fc0937fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:21.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.778+0000 7fc0b12d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:21.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.778+0000 7fc0b12d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:21.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.778+0000 7fc0b12d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:21.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.778+0000 7fc0b12d9640 1 -- 172.21.15.18:0/860096116 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0ac1434a0 con 0x7fc0ac1430c0 2024-03-25T18:05:21.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.814+0000 7fc0937fe640 1 -- 172.21.15.18:0/860096116 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fc0a40510a0 con 0x7fc0ac1430c0 2024-03-25T18:05:21.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.814+0000 7fc0937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:21.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.814+0000 7fc0937fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:21.816 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:05:21.816 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:21.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.817+0000 7fc0b12d9640 10 librados: watch_flush enter 2024-03-25T18:05:21.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.817+0000 7fc0b12d9640 10 librados: watch_flush exit 2024-03-25T18:05:21.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.817+0000 7fc0b12d9640 20 client.4635.objecter shutdown clearing up homeless session... 2024-03-25T18:05:21.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.817+0000 7fc0b12d9640 10 client.4635.objecter successfully canceled tick 2024-03-25T18:05:21.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.817+0000 7fc0b12d9640 1 -- 172.21.15.18:0/860096116 >> v1:172.21.15.18:6800/3721244451 conn(0x7fc09806dd90 legacy=0x7fc098070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:21.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.817+0000 7fc0b12d9640 10 monclient: shutdown 2024-03-25T18:05:21.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.817+0000 7fc0b12d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:21.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.817+0000 7fc0b12d9640 1 -- 172.21.15.18:0/860096116 >> v1:172.21.15.18:6789/0 conn(0x7fc0ac1430c0 legacy=0x7fc0ac1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:21.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.817+0000 7fc0b12d9640 1 -- 172.21.15.18:0/860096116 shutdown_connections 2024-03-25T18:05:21.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.817+0000 7fc0b12d9640 1 -- 172.21.15.18:0/860096116 >> 172.21.15.18:0/860096116 conn(0x7fc0ac0b7ea0 msgr2=0x7fc0ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:21.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.817+0000 7fc0b12d9640 1 -- 172.21.15.18:0/860096116 shutdown_connections 2024-03-25T18:05:21.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.817+0000 7fc0b12d9640 1 -- 172.21.15.18:0/860096116 wait complete. 2024-03-25T18:05:21.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:21.817+0000 7fc0b12d9640 1 librados: shutdown 2024-03-25T18:05:21.830 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:28.636 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:28.752 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.750+0000 7f3fedae4640 10 monclient: get_monmap_and_config 2024-03-25T18:05:28.752 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.750+0000 7f3fedae4640 10 monclient: build_initial_monmap 2024-03-25T18:05:28.752 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.751+0000 7f3fedae4640 10 monclient: monmap: 2024-03-25T18:05:28.752 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:05:28.752 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:28.752 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:05:28.752382+0000 2024-03-25T18:05:28.753 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:05:28.752382+0000 2024-03-25T18:05:28.753 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:28.753 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:28.753 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:05:28.753 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:05:28.753 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:05:28.753 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:28.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.751+0000 7f3fedae4640 1 Processor -- start 2024-03-25T18:05:28.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.751+0000 7f3fedae4640 1 -- start start 2024-03-25T18:05:28.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.751+0000 7f3fedae4640 10 monclient: init 2024-03-25T18:05:28.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.752+0000 7f3fedae4640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:28.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.752+0000 7f3fedae4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:05:28.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.752+0000 7f3fedae4640 10 monclient(hunting): picked mon.noname-b con 0x7f3fe80a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:05:28.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.752+0000 7f3fedae4640 10 monclient(hunting): picked mon.noname-a con 0x7f3fe813f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:05:28.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.752+0000 7f3fedae4640 10 monclient(hunting): picked mon.noname-c con 0x7f3fe81430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:05:28.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.752+0000 7f3fedae4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe8142520 con 0x7f3fe813f710 2024-03-25T18:05:28.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.752+0000 7f3fedae4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe80b28a0 con 0x7f3fe81430c0 2024-03-25T18:05:28.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.752+0000 7f3fedae4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe80b2a30 con 0x7f3fe80a98f0 2024-03-25T18:05:28.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.752+0000 7f3fedae4640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:28.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.752+0000 7f3fedae4640 10 monclient(hunting): authenticate will time out at 7457.331376s 2024-03-25T18:05:28.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.761+0000 7f3fe7fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f3fe81430c0 0x7f3fe814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:39564/0 (socket says 172.21.15.18:39564) 2024-03-25T18:05:28.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.761+0000 7f3fe7fff640 1 -- 172.21.15.18:0/3381774862 learned_addr learned my addr 172.21.15.18:0/3381774862 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:05:28.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.777+0000 7f3fe67fc640 1 -- 172.21.15.18:0/3381774862 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489301919 0 0) 0x7f3fe80b28a0 con 0x7f3fe81430c0 2024-03-25T18:05:28.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.777+0000 7f3fe67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:28.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.777+0000 7f3fe67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:28.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.777+0000 7f3fe67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:28.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.777+0000 7f3fe67fc640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:05:28.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.777+0000 7f3fe67fc640 1 -- 172.21.15.18:0/3381774862 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd0003520 con 0x7f3fe81430c0 2024-03-25T18:05:28.785 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.783+0000 7f3fe67fc640 1 -- 172.21.15.18:0/3381774862 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3267050990 0 0) 0x7f3fd0003520 con 0x7f3fe81430c0 2024-03-25T18:05:28.785 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.783+0000 7f3fe67fc640 1 -- 172.21.15.18:0/3381774862 >> v1:172.21.15.18:6790/0 conn(0x7f3fe80a98f0 legacy=0x7f3fe80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:28.785 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.783+0000 7f3fe67fc640 1 -- 172.21.15.18:0/3381774862 >> v1:172.21.15.18:6789/0 conn(0x7f3fe813f710 legacy=0x7f3fe81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:28.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.783+0000 7f3fe67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:05:28.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.783+0000 7f3fe67fc640 1 monclient: found mon.noname-c 2024-03-25T18:05:28.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.783+0000 7f3fe67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:28.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.783+0000 7f3fe67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:05:28.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.783+0000 7f3fe67fc640 1 -- 172.21.15.18:0/3381774862 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe80b44c0 con 0x7f3fe81430c0 2024-03-25T18:05:28.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.783+0000 7f3fe67fc640 10 monclient: _finish_auth 0 2024-03-25T18:05:28.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.783+0000 7f3fe67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:05:28.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.783+0000 7f3fe67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:28.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.783+0000 7f3fe67fc640 1 -- 172.21.15.18:0/3381774862 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3fe0001a10 con 0x7f3fe81430c0 2024-03-25T18:05:28.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.783+0000 7f3fedae4640 5 monclient: authenticate success, global_id 4624 2024-03-25T18:05:28.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.783+0000 7f3fedae4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:28.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.783+0000 7f3fe67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:28.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.784+0000 7f3fe67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:28.787 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.784+0000 7f3fe67fc640 10 monclient: dump: 2024-03-25T18:05:28.787 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:28.787 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:28.787 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:28.787 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:28.787 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:28.787 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:28.787 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:28.787 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:28.787 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:28.787 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:28.787 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.784+0000 7f3fedae4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:28.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.792+0000 7f3fe67fc640 1 -- 172.21.15.18:0/3381774862 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fe0001b70 con 0x7f3fe81430c0 2024-03-25T18:05:28.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.792+0000 7f3fe67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:28.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.792+0000 7f3fe67fc640 1 -- 172.21.15.18:0/3381774862 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3fe0001f80 con 0x7f3fe81430c0 2024-03-25T18:05:28.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.792+0000 7f3fedae4640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:28.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.792+0000 7f3fedae4640 4 set_mon_vals no callback set 2024-03-25T18:05:28.793 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.792+0000 7f3fe67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:28.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.792+0000 7f3fe67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:28.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.792+0000 7f3fe67fc640 10 monclient: dump: 2024-03-25T18:05:28.794 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:28.794 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:28.794 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:28.794 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:28.794 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:28.794 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:28.794 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:28.794 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:28.794 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:28.794 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:28.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.793+0000 7f3fedae4640 10 monclient: shutdown 2024-03-25T18:05:28.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.793+0000 7f3fedae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:28.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.793+0000 7f3fedae4640 1 -- 172.21.15.18:0/3381774862 >> v1:172.21.15.169:6789/0 conn(0x7f3fe81430c0 legacy=0x7f3fe814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:28.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.793+0000 7f3fedae4640 1 -- 172.21.15.18:0/3381774862 shutdown_connections 2024-03-25T18:05:28.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.793+0000 7f3fedae4640 1 -- 172.21.15.18:0/3381774862 >> 172.21.15.18:0/3381774862 conn(0x7f3fe80b7ea0 msgr2=0x7f3fe80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:28.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.793+0000 7f3fedae4640 1 -- 172.21.15.18:0/3381774862 shutdown_connections 2024-03-25T18:05:28.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.793+0000 7f3fedae4640 1 -- 172.21.15.18:0/3381774862 wait complete. 2024-03-25T18:05:28.796 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.794+0000 7f3fedae4640 10 monclient: build_initial_monmap 2024-03-25T18:05:28.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.794+0000 7f3fedae4640 10 monclient: monmap: 2024-03-25T18:05:28.797 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:05:28.797 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:28.797 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:05:28.797 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:05:28.797 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:28.797 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:05:28.797 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:05:28.797 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:05:28.797 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:05:28.797 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:28.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.794+0000 7f3fedae4640 1 librados: starting msgr at 2024-03-25T18:05:28.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.794+0000 7f3fedae4640 1 librados: starting objecter 2024-03-25T18:05:28.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.794+0000 7f3fedae4640 1 Processor -- start 2024-03-25T18:05:28.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.795+0000 7f3fedae4640 1 -- start start 2024-03-25T18:05:28.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.795+0000 7f3fedae4640 1 librados: setting wanted keys 2024-03-25T18:05:28.798 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.795+0000 7f3fedae4640 1 librados: calling monclient init 2024-03-25T18:05:28.798 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.795+0000 7f3fedae4640 10 monclient: init 2024-03-25T18:05:28.798 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.795+0000 7f3fedae4640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:28.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.795+0000 7f3fedae4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:05:28.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.795+0000 7f3fedae4640 10 monclient(hunting): picked mon.noname-a con 0x7f3fe80a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:05:28.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.795+0000 7f3fedae4640 10 monclient(hunting): picked mon.noname-b con 0x7f3fe813f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:05:28.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.795+0000 7f3fedae4640 10 monclient(hunting): picked mon.noname-c con 0x7f3fe81430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:05:28.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.795+0000 7f3fedae4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe81d8340 con 0x7f3fe80a98f0 2024-03-25T18:05:28.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.795+0000 7f3fedae4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe81d7710 con 0x7f3fe813f710 2024-03-25T18:05:28.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.795+0000 7f3fedae4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe8138e90 con 0x7f3fe81430c0 2024-03-25T18:05:28.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.795+0000 7f3fedae4640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:28.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.795+0000 7f3fedae4640 10 monclient(hunting): authenticate will time out at 7457.374561s 2024-03-25T18:05:28.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.804+0000 7f3fe7fff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f3fe81430c0 0x7f3fe8133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:49108/0 (socket says 172.21.15.18:49108) 2024-03-25T18:05:28.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.804+0000 7f3fe7fff640 1 -- 172.21.15.18:0/2973090086 learned_addr learned my addr 172.21.15.18:0/2973090086 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:05:28.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.805+0000 7f3fcffff640 10 client.?.objecter ms_handle_connect 0x7f3fe81430c0 2024-03-25T18:05:28.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.805+0000 7f3fcffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:28.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.809+0000 7f3fcffff640 10 client.?.objecter ms_handle_connect 0x7f3fe813f710 2024-03-25T18:05:28.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.809+0000 7f3fcffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:28.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.817+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959669382 0 0) 0x7f3fe81d7710 con 0x7f3fe813f710 2024-03-25T18:05:28.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.817+0000 7f3fcffff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:28.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.817+0000 7f3fcffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:28.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.817+0000 7f3fcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:28.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.817+0000 7f3fcffff640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:05:28.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.817+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd4004eb0 con 0x7f3fe813f710 2024-03-25T18:05:28.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.833+0000 7f3fcffff640 10 client.?.objecter ms_handle_connect 0x7f3fe80a98f0 2024-03-25T18:05:28.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.833+0000 7f3fcffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:28.879 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:26.671 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:28.886 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:26.678 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:28.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.904+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021328034 0 0) 0x7f3fe81d8340 con 0x7f3fe80a98f0 2024-03-25T18:05:28.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.904+0000 7f3fcffff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:28.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.904+0000 7f3fcffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:28.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.904+0000 7f3fcffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:28.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.904+0000 7f3fcffff640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:05:28.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.904+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe81d7710 con 0x7f3fe80a98f0 2024-03-25T18:05:28.918 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:26.711 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:28.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.935+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2607482676 0 0) 0x7f3fe81d7710 con 0x7f3fe80a98f0 2024-03-25T18:05:28.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.935+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fe81d8340 con 0x7f3fe80a98f0 2024-03-25T18:05:28.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.935+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3fd80018a0 con 0x7f3fe80a98f0 2024-03-25T18:05:28.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.935+0000 7f3fcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:28.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.935+0000 7f3fcffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:28.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.935+0000 7f3fcffff640 10 monclient(hunting): dump: 2024-03-25T18:05:28.937 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:28.937 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:28.937 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:28.937 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:28.937 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:28.937 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:28.937 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:28.937 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:28.937 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:28.938 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:28.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.935+0000 7f3fcffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:28.946 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:26.695 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:28.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.958+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1327953309 0 0) 0x7f3fd4004eb0 con 0x7f3fe813f710 2024-03-25T18:05:28.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.958+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fe81d7710 con 0x7f3fe813f710 2024-03-25T18:05:28.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.958+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3fdc001930 con 0x7f3fe813f710 2024-03-25T18:05:28.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.958+0000 7f3fcffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:28.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.958+0000 7f3fcffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:28.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.958+0000 7f3fcffff640 10 monclient(hunting): dump: 2024-03-25T18:05:28.960 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:28.960 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:28.960 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:28.960 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:28.960 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:28.960 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:28.960 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:28.960 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:28.960 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:28.960 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:28.962 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:26.710 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:28.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.964+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3459769547 0 0) 0x7f3fe81d8340 con 0x7f3fe80a98f0 2024-03-25T18:05:28.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.964+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 >> v1:172.21.15.18:6790/0 conn(0x7f3fe81430c0 legacy=0x7f3fe8133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:28.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.964+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 >> v1:172.21.15.169:6789/0 conn(0x7f3fe813f710 legacy=0x7f3fe8138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:28.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.964+0000 7f3fcffff640 10 monclient: _finish_hunting 0 2024-03-25T18:05:28.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.964+0000 7f3fcffff640 1 monclient: found mon.a 2024-03-25T18:05:28.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.964+0000 7f3fcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:28.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.964+0000 7f3fcffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:28.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.964+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe8139020 con 0x7f3fe80a98f0 2024-03-25T18:05:28.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.964+0000 7f3fcffff640 10 monclient: _finish_auth 0 2024-03-25T18:05:28.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.964+0000 7f3fcffff640 10 monclient: _check_auth_tickets 2024-03-25T18:05:28.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.964+0000 7f3fcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:28.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.964+0000 7f3fedae4640 5 monclient: authenticate success, global_id 4638 2024-03-25T18:05:28.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.964+0000 7f3fedae4640 10 monclient: _renew_subs 2024-03-25T18:05:28.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.964+0000 7f3fedae4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:28.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.964+0000 7f3fedae4640 1 -- 172.21.15.18:0/2973090086 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe8137f30 con 0x7f3fe80a98f0 2024-03-25T18:05:28.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.964+0000 7f3fedae4640 10 client.4638.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:28.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.965+0000 7f3fedae4640 10 monclient: _renew_subs 2024-03-25T18:05:28.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.965+0000 7f3fedae4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:28.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.965+0000 7f3fedae4640 1 -- 172.21.15.18:0/2973090086 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe81352b0 con 0x7f3fe80a98f0 2024-03-25T18:05:28.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.965+0000 7f3fedae4640 1 librados: init done 2024-03-25T18:05:28.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.965+0000 7f3fedae4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:28.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.966+0000 7f3fedae4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:28.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.966+0000 7f3fedae4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:28.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:28.966+0000 7f3fedae4640 1 -- 172.21.15.18:0/2973090086 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe81d7500 con 0x7f3fe80a98f0 2024-03-25T18:05:28.969 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:26.762 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:28.974 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:26.767 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:28.993 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:26.741 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:29.001 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:26.794 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:29.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.304+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fd8001d50 con 0x7f3fe80a98f0 2024-03-25T18:05:29.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.304+0000 7f3fcffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:29.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.304+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f3fd8003790 con 0x7f3fe80a98f0 2024-03-25T18:05:29.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.304+0000 7f3fcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:29.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.304+0000 7f3fcffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:29.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.304+0000 7f3fcffff640 10 monclient: dump: 2024-03-25T18:05:29.306 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:29.306 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:29.306 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:29.306 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:29.306 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:29.306 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:29.306 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:29.306 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:29.306 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:29.306 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:29.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.304+0000 7f3fe5ffb640 4 set_mon_vals no callback set 2024-03-25T18:05:29.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.305+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f3fd80039b0 con 0x7f3fe80a98f0 2024-03-25T18:05:29.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.305+0000 7f3fe6ffd640 10 monclient: get_auth_request con 0x7f3fd406de90 auth_method 0 2024-03-25T18:05:29.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.306+0000 7f3fcffff640 10 client.4638.objecter ms_handle_connect 0x7f3fd406de90 2024-03-25T18:05:29.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.312+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f3fd8053510 con 0x7f3fe80a98f0 2024-03-25T18:05:29.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.312+0000 7f3fcffff640 10 client.4638.objecter ms_dispatch 0x7f3fe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:29.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.312+0000 7f3fcffff640 3 client.4638.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:29.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.312+0000 7f3fcffff640 3 client.4638.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:29.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.312+0000 7f3fcffff640 20 client.4638.objecter dump_active .. 0 homeless 2024-03-25T18:05:29.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.312+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3fd804ee50 con 0x7f3fe80a98f0 2024-03-25T18:05:29.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.312+0000 7f3fcffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:29.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.312+0000 7f3fcffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:29.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.557+0000 7f3fedae4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:29.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.557+0000 7f3fedae4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:29.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.557+0000 7f3fedae4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:29.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.557+0000 7f3fedae4640 1 -- 172.21.15.18:0/2973090086 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fe80627b0 con 0x7f3fe80a98f0 2024-03-25T18:05:29.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.590+0000 7f3fcffff640 1 -- 172.21.15.18:0/2973090086 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f3fd8050f50 con 0x7f3fe80a98f0 2024-03-25T18:05:29.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.590+0000 7f3fcffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:29.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.590+0000 7f3fcffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:29.592 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:05:29.592 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:29.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.592+0000 7f3fedae4640 10 librados: watch_flush enter 2024-03-25T18:05:29.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.592+0000 7f3fedae4640 10 librados: watch_flush exit 2024-03-25T18:05:29.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.592+0000 7f3fedae4640 20 client.4638.objecter shutdown clearing up homeless session... 2024-03-25T18:05:29.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.592+0000 7f3fedae4640 10 client.4638.objecter successfully canceled tick 2024-03-25T18:05:29.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.593+0000 7f3fedae4640 1 -- 172.21.15.18:0/2973090086 >> v1:172.21.15.18:6800/3721244451 conn(0x7f3fd406de90 legacy=0x7f3fd4070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:29.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.593+0000 7f3fedae4640 10 monclient: shutdown 2024-03-25T18:05:29.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.593+0000 7f3fedae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:29.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.593+0000 7f3fedae4640 1 -- 172.21.15.18:0/2973090086 >> v1:172.21.15.18:6789/0 conn(0x7f3fe80a98f0 legacy=0x7f3fe81d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:29.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.593+0000 7f3fedae4640 1 -- 172.21.15.18:0/2973090086 shutdown_connections 2024-03-25T18:05:29.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.593+0000 7f3fedae4640 1 -- 172.21.15.18:0/2973090086 >> 172.21.15.18:0/2973090086 conn(0x7f3fe80b7ea0 msgr2=0x7f3fe8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:29.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.593+0000 7f3fedae4640 1 -- 172.21.15.18:0/2973090086 shutdown_connections 2024-03-25T18:05:29.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.593+0000 7f3fedae4640 1 -- 172.21.15.18:0/2973090086 wait complete. 2024-03-25T18:05:29.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:29.593+0000 7f3fedae4640 1 librados: shutdown 2024-03-25T18:05:29.607 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:36.413 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:36.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.529+0000 7fda9f9b6640 10 monclient: get_monmap_and_config 2024-03-25T18:05:36.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.529+0000 7fda9f9b6640 10 monclient: build_initial_monmap 2024-03-25T18:05:36.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.529+0000 7fda9f9b6640 10 monclient: monmap: 2024-03-25T18:05:36.530 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:05:36.531 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:36.531 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:05:36.530512+0000 2024-03-25T18:05:36.531 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:05:36.530512+0000 2024-03-25T18:05:36.531 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:36.531 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:36.531 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:05:36.531 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:05:36.531 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:05:36.531 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:36.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.529+0000 7fda9f9b6640 1 Processor -- start 2024-03-25T18:05:36.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.530+0000 7fda9f9b6640 1 -- start start 2024-03-25T18:05:36.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.530+0000 7fda9f9b6640 10 monclient: init 2024-03-25T18:05:36.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.530+0000 7fda9f9b6640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:36.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.530+0000 7fda9f9b6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:05:36.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.530+0000 7fda9f9b6640 10 monclient(hunting): picked mon.noname-a con 0x7fda980b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T18:05:36.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.530+0000 7fda9f9b6640 10 monclient(hunting): picked mon.noname-c con 0x7fda980b1730 addr v1:172.21.15.169:6789/0 2024-03-25T18:05:36.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.530+0000 7fda9f9b6640 10 monclient(hunting): picked mon.noname-b con 0x7fda9814be90 addr v1:172.21.15.18:6790/0 2024-03-25T18:05:36.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.530+0000 7fda9f9b6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda980af940 con 0x7fda980b4d60 2024-03-25T18:05:36.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.530+0000 7fda9f9b6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda980b0a70 con 0x7fda980b1730 2024-03-25T18:05:36.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.530+0000 7fda9f9b6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda980b0d50 con 0x7fda9814be90 2024-03-25T18:05:36.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.531+0000 7fda9f9b6640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:36.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.531+0000 7fda9f9b6640 10 monclient(hunting): authenticate will time out at 7465.109658s 2024-03-25T18:05:36.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.533+0000 7fda9df53640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fda9814be90 0x7fda9814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:59976/0 (socket says 172.21.15.18:59976) 2024-03-25T18:05:36.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.533+0000 7fda9df53640 1 -- 172.21.15.18:0/2432817048 learned_addr learned my addr 172.21.15.18:0/2432817048 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:05:36.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.556+0000 7fda8ffff640 1 -- 172.21.15.18:0/2432817048 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993528790 0 0) 0x7fda980b0a70 con 0x7fda980b1730 2024-03-25T18:05:36.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.556+0000 7fda8ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:36.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.556+0000 7fda8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:36.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.556+0000 7fda8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:36.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.556+0000 7fda8ffff640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:05:36.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.556+0000 7fda8ffff640 1 -- 172.21.15.18:0/2432817048 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda84003520 con 0x7fda980b1730 2024-03-25T18:05:36.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.559+0000 7fda8ffff640 1 -- 172.21.15.18:0/2432817048 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161744359 0 0) 0x7fda980b0d50 con 0x7fda9814be90 2024-03-25T18:05:36.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.559+0000 7fda8ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:36.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.559+0000 7fda8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:36.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.559+0000 7fda8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:36.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.559+0000 7fda8ffff640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:05:36.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.559+0000 7fda8ffff640 1 -- 172.21.15.18:0/2432817048 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda980b0a70 con 0x7fda9814be90 2024-03-25T18:05:36.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.565+0000 7fda8ffff640 1 -- 172.21.15.18:0/2432817048 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716164791 0 0) 0x7fda980af940 con 0x7fda980b4d60 2024-03-25T18:05:36.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.565+0000 7fda8ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:36.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.565+0000 7fda8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:36.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.565+0000 7fda8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:36.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.565+0000 7fda8ffff640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:05:36.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.565+0000 7fda8ffff640 1 -- 172.21.15.18:0/2432817048 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda980b0d50 con 0x7fda980b4d60 2024-03-25T18:05:36.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.573+0000 7fda8ffff640 1 -- 172.21.15.18:0/2432817048 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3309699268 0 0) 0x7fda84003520 con 0x7fda980b1730 2024-03-25T18:05:36.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.573+0000 7fda8ffff640 1 -- 172.21.15.18:0/2432817048 >> v1:172.21.15.18:6790/0 conn(0x7fda9814be90 legacy=0x7fda9814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:36.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.573+0000 7fda8ffff640 1 -- 172.21.15.18:0/2432817048 >> v1:172.21.15.18:6789/0 conn(0x7fda980b4d60 legacy=0x7fda9813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:36.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.573+0000 7fda8ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:05:36.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.573+0000 7fda8ffff640 1 monclient: found mon.noname-c 2024-03-25T18:05:36.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.573+0000 7fda8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:36.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.573+0000 7fda8ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:05:36.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.573+0000 7fda8ffff640 1 -- 172.21.15.18:0/2432817048 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda980b1030 con 0x7fda980b1730 2024-03-25T18:05:36.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.573+0000 7fda8ffff640 10 monclient: _finish_auth 0 2024-03-25T18:05:36.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.573+0000 7fda8ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:05:36.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.573+0000 7fda8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:36.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.573+0000 7fda8ffff640 1 -- 172.21.15.18:0/2432817048 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fda88001690 con 0x7fda980b1730 2024-03-25T18:05:36.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.573+0000 7fda9f9b6640 5 monclient: authenticate success, global_id 4630 2024-03-25T18:05:36.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.573+0000 7fda8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:36.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.573+0000 7fda8ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:36.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.573+0000 7fda8ffff640 10 monclient: dump: 2024-03-25T18:05:36.576 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:36.576 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:36.576 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:36.576 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:36.577 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:36.577 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:36.577 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:36.577 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:36.577 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:36.577 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:36.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.573+0000 7fda9f9b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:36.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.577+0000 7fda8ffff640 1 -- 172.21.15.18:0/2432817048 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda880017f0 con 0x7fda980b1730 2024-03-25T18:05:36.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.577+0000 7fda8ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:36.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.577+0000 7fda8ffff640 1 -- 172.21.15.18:0/2432817048 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fda88001c00 con 0x7fda980b1730 2024-03-25T18:05:36.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.577+0000 7fda9f9b6640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:36.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.577+0000 7fda9f9b6640 4 set_mon_vals no callback set 2024-03-25T18:05:36.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.577+0000 7fda8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:36.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.577+0000 7fda8ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:36.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.577+0000 7fda8ffff640 10 monclient: dump: 2024-03-25T18:05:36.579 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:36.579 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:36.579 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:36.579 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:36.579 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:36.579 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:36.579 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:36.579 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:36.580 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:36.580 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:36.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.577+0000 7fda9f9b6640 10 monclient: shutdown 2024-03-25T18:05:36.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.577+0000 7fda9f9b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:36.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.577+0000 7fda9f9b6640 1 -- 172.21.15.18:0/2432817048 >> v1:172.21.15.169:6789/0 conn(0x7fda980b1730 legacy=0x7fda980b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:36.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.577+0000 7fda9f9b6640 1 -- 172.21.15.18:0/2432817048 shutdown_connections 2024-03-25T18:05:36.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.577+0000 7fda9f9b6640 1 -- 172.21.15.18:0/2432817048 >> 172.21.15.18:0/2432817048 conn(0x7fda980617a0 msgr2=0x7fda98061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:36.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.578+0000 7fda9f9b6640 1 -- 172.21.15.18:0/2432817048 shutdown_connections 2024-03-25T18:05:36.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.578+0000 7fda9f9b6640 1 -- 172.21.15.18:0/2432817048 wait complete. 2024-03-25T18:05:36.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.578+0000 7fda9f9b6640 10 monclient: build_initial_monmap 2024-03-25T18:05:36.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.578+0000 7fda9f9b6640 10 monclient: monmap: 2024-03-25T18:05:36.582 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:05:36.582 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:36.583 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:05:36.583 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:05:36.583 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:36.583 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:05:36.583 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:05:36.583 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:05:36.583 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:05:36.583 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:36.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.578+0000 7fda9f9b6640 1 librados: starting msgr at 2024-03-25T18:05:36.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.578+0000 7fda9f9b6640 1 librados: starting objecter 2024-03-25T18:05:36.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.579+0000 7fda9f9b6640 1 Processor -- start 2024-03-25T18:05:36.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.579+0000 7fda9f9b6640 1 -- start start 2024-03-25T18:05:36.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.579+0000 7fda9f9b6640 1 librados: setting wanted keys 2024-03-25T18:05:36.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.579+0000 7fda9f9b6640 1 librados: calling monclient init 2024-03-25T18:05:36.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.579+0000 7fda9f9b6640 10 monclient: init 2024-03-25T18:05:36.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.579+0000 7fda9f9b6640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:36.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.579+0000 7fda9f9b6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:05:36.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.579+0000 7fda9f9b6640 10 monclient(hunting): picked mon.noname-c con 0x7fda980b1730 addr v1:172.21.15.18:6790/0 2024-03-25T18:05:36.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.579+0000 7fda9f9b6640 10 monclient(hunting): picked mon.noname-a con 0x7fda980b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T18:05:36.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.579+0000 7fda9f9b6640 10 monclient(hunting): picked mon.noname-b con 0x7fda9814be90 addr v1:172.21.15.169:6789/0 2024-03-25T18:05:36.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.579+0000 7fda9f9b6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda9813adc0 con 0x7fda980b4d60 2024-03-25T18:05:36.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.579+0000 7fda9f9b6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda9813b0a0 con 0x7fda9814be90 2024-03-25T18:05:36.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.579+0000 7fda9f9b6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda9813bbd0 con 0x7fda980b1730 2024-03-25T18:05:36.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.579+0000 7fda9f9b6640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:36.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.579+0000 7fda9f9b6640 10 monclient(hunting): authenticate will time out at 7465.158505s 2024-03-25T18:05:36.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.588+0000 7fda9df53640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fda9814be90 0x7fda981e2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:55860/0 (socket says 172.21.15.18:55860) 2024-03-25T18:05:36.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.588+0000 7fda9df53640 1 -- 172.21.15.18:0/2375916555 learned_addr learned my addr 172.21.15.18:0/2375916555 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:05:36.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.589+0000 7fda8dffb640 10 client.?.objecter ms_handle_connect 0x7fda9814be90 2024-03-25T18:05:36.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.589+0000 7fda8dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:36.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.597+0000 7fda8dffb640 10 client.?.objecter ms_handle_connect 0x7fda980b4d60 2024-03-25T18:05:36.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.597+0000 7fda8dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:36.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.607+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483497261 0 0) 0x7fda9813adc0 con 0x7fda980b4d60 2024-03-25T18:05:36.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.607+0000 7fda8dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:36.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.607+0000 7fda8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:36.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.607+0000 7fda8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:36.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.607+0000 7fda8dffb640 10 monclient(hunting): my global_id is 4644 2024-03-25T18:05:36.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.607+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda7c004eb0 con 0x7fda980b4d60 2024-03-25T18:05:36.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.608+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123838109 0 0) 0x7fda9813b0a0 con 0x7fda9814be90 2024-03-25T18:05:36.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.608+0000 7fda8dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:36.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.608+0000 7fda8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:36.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.608+0000 7fda8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:36.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.608+0000 7fda8dffb640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:05:36.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.608+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda9813adc0 con 0x7fda9814be90 2024-03-25T18:05:36.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.615+0000 7fda8dffb640 10 client.?.objecter ms_handle_connect 0x7fda980b1730 2024-03-25T18:05:36.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.615+0000 7fda8dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:36.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.618+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 114860777 0 0) 0x7fda9813adc0 con 0x7fda9814be90 2024-03-25T18:05:36.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.618+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda9813b0a0 con 0x7fda9814be90 2024-03-25T18:05:36.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.618+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fda90001c50 con 0x7fda9814be90 2024-03-25T18:05:36.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.618+0000 7fda8dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:36.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.618+0000 7fda8dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:36.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.618+0000 7fda8dffb640 10 monclient(hunting): dump: 2024-03-25T18:05:36.620 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:36.621 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:36.621 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:36.621 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:36.621 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:36.621 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:36.621 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:36.621 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:36.621 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:36.621 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:36.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.618+0000 7fda8dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:36.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.621+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 774861533 0 0) 0x7fda7c004eb0 con 0x7fda980b4d60 2024-03-25T18:05:36.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.621+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda9813adc0 con 0x7fda980b4d60 2024-03-25T18:05:36.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.621+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fda88001b00 con 0x7fda980b4d60 2024-03-25T18:05:36.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.621+0000 7fda8dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:36.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.621+0000 7fda8dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:36.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.621+0000 7fda8dffb640 10 monclient(hunting): dump: 2024-03-25T18:05:36.623 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:36.623 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:36.623 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:36.623 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:36.623 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:36.624 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:36.624 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:36.624 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:36.624 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:36.624 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:36.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.634+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121955336 0 0) 0x7fda9813bbd0 con 0x7fda980b1730 2024-03-25T18:05:36.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.634+0000 7fda8dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:36.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.634+0000 7fda8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:36.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.634+0000 7fda8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:36.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.634+0000 7fda8dffb640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:05:36.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.634+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda7c004eb0 con 0x7fda980b1730 2024-03-25T18:05:36.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.637+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3881335291 0 0) 0x7fda9813b0a0 con 0x7fda9814be90 2024-03-25T18:05:36.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 >> v1:172.21.15.18:6790/0 conn(0x7fda980b1730 legacy=0x7fda98139270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:36.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 >> v1:172.21.15.18:6789/0 conn(0x7fda980b4d60 legacy=0x7fda9813b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:36.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda8dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:05:36.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda8dffb640 1 monclient: found mon.b 2024-03-25T18:05:36.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda8dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:36.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda8dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:05:36.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda9813bd60 con 0x7fda9814be90 2024-03-25T18:05:36.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda8dffb640 10 monclient: _finish_auth 0 2024-03-25T18:05:36.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda8dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:05:36.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:36.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda9f9b6640 5 monclient: authenticate success, global_id 4633 2024-03-25T18:05:36.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda9f9b6640 10 monclient: _renew_subs 2024-03-25T18:05:36.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda9f9b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:05:36.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda9f9b6640 1 -- 172.21.15.18:0/2375916555 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda98143150 con 0x7fda9814be90 2024-03-25T18:05:36.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda9f9b6640 10 client.4633.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:36.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda9f9b6640 10 monclient: _renew_subs 2024-03-25T18:05:36.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda9f9b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:05:36.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda9f9b6640 1 -- 172.21.15.18:0/2375916555 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda98139f80 con 0x7fda9814be90 2024-03-25T18:05:36.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.638+0000 7fda9f9b6640 1 librados: init done 2024-03-25T18:05:36.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.639+0000 7fda9f9b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:36.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.639+0000 7fda9f9b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:36.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.639+0000 7fda9f9b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:05:36.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.639+0000 7fda9f9b6640 1 -- 172.21.15.18:0/2375916555 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda980b58b0 con 0x7fda9814be90 2024-03-25T18:05:36.960 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.959+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda90004b60 con 0x7fda9814be90 2024-03-25T18:05:36.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.959+0000 7fda8dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:36.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.959+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fda90004e70 con 0x7fda9814be90 2024-03-25T18:05:36.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.959+0000 7fda8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:36.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.959+0000 7fda8dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:36.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.959+0000 7fda8dffb640 10 monclient: dump: 2024-03-25T18:05:36.961 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:36.961 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:36.961 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:36.961 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:36.961 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:36.961 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:36.961 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:36.962 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:36.962 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:36.962 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:36.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.959+0000 7fda8f7fe640 4 set_mon_vals no callback set 2024-03-25T18:05:36.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.960+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fda9001ae40 con 0x7fda9814be90 2024-03-25T18:05:36.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.960+0000 7fda9d752640 10 monclient: get_auth_request con 0x7fda7c06df50 auth_method 0 2024-03-25T18:05:36.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.960+0000 7fda8dffb640 10 client.4633.objecter ms_handle_connect 0x7fda7c06df50 2024-03-25T18:05:36.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.965+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fda90054a00 con 0x7fda9814be90 2024-03-25T18:05:36.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.965+0000 7fda8dffb640 10 client.4633.objecter ms_dispatch 0x7fda98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:36.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.965+0000 7fda8dffb640 3 client.4633.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:36.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.965+0000 7fda8dffb640 3 client.4633.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:36.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.965+0000 7fda8dffb640 20 client.4633.objecter dump_active .. 0 homeless 2024-03-25T18:05:36.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.965+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fda90050340 con 0x7fda9814be90 2024-03-25T18:05:36.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.965+0000 7fda8dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:36.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:36.965+0000 7fda8dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:37.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.190+0000 7fda9f9b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:37.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.190+0000 7fda9f9b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:37.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.190+0000 7fda9f9b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:05:37.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.190+0000 7fda9f9b6640 1 -- 172.21.15.18:0/2375916555 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda980b5f40 con 0x7fda9814be90 2024-03-25T18:05:37.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.218+0000 7fda8dffb640 1 -- 172.21.15.18:0/2375916555 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fda90052440 con 0x7fda9814be90 2024-03-25T18:05:37.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.218+0000 7fda8dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:37.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.218+0000 7fda8dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:37.219 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:05:37.220 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:37.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.221+0000 7fda9f9b6640 10 librados: watch_flush enter 2024-03-25T18:05:37.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.221+0000 7fda9f9b6640 10 librados: watch_flush exit 2024-03-25T18:05:37.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.221+0000 7fda9f9b6640 20 client.4633.objecter shutdown clearing up homeless session... 2024-03-25T18:05:37.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.221+0000 7fda9f9b6640 10 client.4633.objecter successfully canceled tick 2024-03-25T18:05:37.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.221+0000 7fda9f9b6640 1 -- 172.21.15.18:0/2375916555 >> v1:172.21.15.18:6800/3721244451 conn(0x7fda7c06df50 legacy=0x7fda7c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:37.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.221+0000 7fda9f9b6640 10 monclient: shutdown 2024-03-25T18:05:37.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.221+0000 7fda9f9b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:37.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.221+0000 7fda9f9b6640 1 -- 172.21.15.18:0/2375916555 >> v1:172.21.15.169:6789/0 conn(0x7fda9814be90 legacy=0x7fda981e2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:37.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.221+0000 7fda9f9b6640 1 -- 172.21.15.18:0/2375916555 shutdown_connections 2024-03-25T18:05:37.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.221+0000 7fda9f9b6640 1 -- 172.21.15.18:0/2375916555 >> 172.21.15.18:0/2375916555 conn(0x7fda980617a0 msgr2=0x7fda980b6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:37.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.221+0000 7fda9f9b6640 1 -- 172.21.15.18:0/2375916555 shutdown_connections 2024-03-25T18:05:37.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.221+0000 7fda9f9b6640 1 -- 172.21.15.18:0/2375916555 wait complete. 2024-03-25T18:05:37.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:37.221+0000 7fda9f9b6640 1 librados: shutdown 2024-03-25T18:05:37.234 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:44.041 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:44.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.149+0000 7fecdd6ba640 10 monclient: get_monmap_and_config 2024-03-25T18:05:44.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.149+0000 7fecdd6ba640 10 monclient: build_initial_monmap 2024-03-25T18:05:44.151 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.149+0000 7fecdd6ba640 10 monclient: monmap: 2024-03-25T18:05:44.151 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:05:44.151 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:44.151 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:05:44.151145+0000 2024-03-25T18:05:44.151 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:05:44.151145+0000 2024-03-25T18:05:44.151 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:44.151 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:44.151 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:05:44.151 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:05:44.152 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:05:44.152 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:44.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.150+0000 7fecdd6ba640 1 Processor -- start 2024-03-25T18:05:44.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.150+0000 7fecdd6ba640 1 -- start start 2024-03-25T18:05:44.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.150+0000 7fecdd6ba640 10 monclient: init 2024-03-25T18:05:44.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.151+0000 7fecdd6ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:44.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.151+0000 7fecdd6ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:05:44.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.151+0000 7fecdd6ba640 10 monclient(hunting): picked mon.noname-b con 0x7fecd80a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:05:44.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.151+0000 7fecdd6ba640 10 monclient(hunting): picked mon.noname-a con 0x7fecd813f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:05:44.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.151+0000 7fecdd6ba640 10 monclient(hunting): picked mon.noname-c con 0x7fecd81430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:05:44.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.151+0000 7fecdd6ba640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd8142520 con 0x7fecd813f710 2024-03-25T18:05:44.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.151+0000 7fecdd6ba640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd80b28a0 con 0x7fecd81430c0 2024-03-25T18:05:44.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.151+0000 7fecdd6ba640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd80b2a30 con 0x7fecd80a98f0 2024-03-25T18:05:44.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.151+0000 7fecdd6ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:44.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.151+0000 7fecdd6ba640 10 monclient(hunting): authenticate will time out at 7472.730067s 2024-03-25T18:05:44.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.157+0000 7fecd6ffd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fecd80a98f0 0x7fecd80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:36070/0 (socket says 172.21.15.18:36070) 2024-03-25T18:05:44.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.157+0000 7fecd6ffd640 1 -- 172.21.15.18:0/734038132 learned_addr learned my addr 172.21.15.18:0/734038132 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:05:44.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.164+0000 7fecd5ffb640 1 -- 172.21.15.18:0/734038132 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176603250 0 0) 0x7fecd80b2a30 con 0x7fecd80a98f0 2024-03-25T18:05:44.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.164+0000 7fecd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:44.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.164+0000 7fecd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:44.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.164+0000 7fecd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:44.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.164+0000 7fecd5ffb640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:05:44.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.164+0000 7fecd5ffb640 1 -- 172.21.15.18:0/734038132 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc0003520 con 0x7fecd80a98f0 2024-03-25T18:05:44.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.174+0000 7fecd5ffb640 1 -- 172.21.15.18:0/734038132 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2258490557 0 0) 0x7fecc0003520 con 0x7fecd80a98f0 2024-03-25T18:05:44.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.174+0000 7fecd5ffb640 1 -- 172.21.15.18:0/734038132 >> v1:172.21.15.169:6789/0 conn(0x7fecd81430c0 legacy=0x7fecd814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:05:44.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.174+0000 7fecd5ffb640 1 -- 172.21.15.18:0/734038132 >> v1:172.21.15.18:6789/0 conn(0x7fecd813f710 legacy=0x7fecd81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:05:44.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.174+0000 7fecd5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:05:44.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.174+0000 7fecd5ffb640 1 monclient: found mon.noname-b 2024-03-25T18:05:44.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.174+0000 7fecd5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:44.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.174+0000 7fecd5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:05:44.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.174+0000 7fecd5ffb640 1 -- 172.21.15.18:0/734038132 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd80b44c0 con 0x7fecd80a98f0 2024-03-25T18:05:44.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.174+0000 7fecd5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:05:44.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.174+0000 7fecd5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:05:44.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.174+0000 7fecd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:44.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.174+0000 7fecd5ffb640 1 -- 172.21.15.18:0/734038132 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fecc8001a10 con 0x7fecd80a98f0 2024-03-25T18:05:44.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.174+0000 7fecd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:44.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.174+0000 7fecd5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:44.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.174+0000 7fecd5ffb640 10 monclient: dump: 2024-03-25T18:05:44.177 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:44.177 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:44.177 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:44.177 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:44.177 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:44.178 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:44.178 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:44.178 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:44.178 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:44.178 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:44.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.174+0000 7fecdd6ba640 5 monclient: authenticate success, global_id 4652 2024-03-25T18:05:44.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.174+0000 7fecdd6ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.186+0000 7fecd5ffb640 1 -- 172.21.15.18:0/734038132 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecc8001be0 con 0x7fecd80a98f0 2024-03-25T18:05:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.186+0000 7fecd5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.186+0000 7fecd5ffb640 1 -- 172.21.15.18:0/734038132 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fecc8002030 con 0x7fecd80a98f0 2024-03-25T18:05:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.186+0000 7fecdd6ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.186+0000 7fecdd6ba640 4 set_mon_vals no callback set 2024-03-25T18:05:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.186+0000 7fecd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.186+0000 7fecd5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.186+0000 7fecd5ffb640 10 monclient: dump: 2024-03-25T18:05:44.188 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:44.188 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:44.189 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:44.189 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:44.189 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:44.189 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:44.189 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:44.189 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:44.189 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:44.189 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:44.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.187+0000 7fecdd6ba640 10 monclient: shutdown 2024-03-25T18:05:44.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.187+0000 7fecdd6ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:44.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.187+0000 7fecdd6ba640 1 -- 172.21.15.18:0/734038132 >> v1:172.21.15.18:6790/0 conn(0x7fecd80a98f0 legacy=0x7fecd80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:44.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.187+0000 7fecdd6ba640 1 -- 172.21.15.18:0/734038132 shutdown_connections 2024-03-25T18:05:44.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.187+0000 7fecdd6ba640 1 -- 172.21.15.18:0/734038132 >> 172.21.15.18:0/734038132 conn(0x7fecd80b7ea0 msgr2=0x7fecd80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:44.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.187+0000 7fecdd6ba640 1 -- 172.21.15.18:0/734038132 shutdown_connections 2024-03-25T18:05:44.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.187+0000 7fecdd6ba640 1 -- 172.21.15.18:0/734038132 wait complete. 2024-03-25T18:05:44.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.188+0000 7fecdd6ba640 10 monclient: build_initial_monmap 2024-03-25T18:05:44.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.188+0000 7fecdd6ba640 10 monclient: monmap: 2024-03-25T18:05:44.192 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:05:44.192 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:44.192 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:05:44.192 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:05:44.192 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:44.192 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:05:44.192 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:05:44.192 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:05:44.192 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:05:44.192 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:44.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.188+0000 7fecdd6ba640 1 librados: starting msgr at 2024-03-25T18:05:44.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.188+0000 7fecdd6ba640 1 librados: starting objecter 2024-03-25T18:05:44.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.188+0000 7fecdd6ba640 1 Processor -- start 2024-03-25T18:05:44.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.189+0000 7fecdd6ba640 1 -- start start 2024-03-25T18:05:44.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.189+0000 7fecdd6ba640 1 librados: setting wanted keys 2024-03-25T18:05:44.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.189+0000 7fecdd6ba640 1 librados: calling monclient init 2024-03-25T18:05:44.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.189+0000 7fecdd6ba640 10 monclient: init 2024-03-25T18:05:44.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.189+0000 7fecdd6ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:44.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.189+0000 7fecdd6ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:05:44.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.189+0000 7fecdd6ba640 10 monclient(hunting): picked mon.noname-a con 0x7fecd80a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:05:44.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.189+0000 7fecdd6ba640 10 monclient(hunting): picked mon.noname-b con 0x7fecd813f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:05:44.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.189+0000 7fecdd6ba640 10 monclient(hunting): picked mon.noname-c con 0x7fecd81430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:05:44.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.189+0000 7fecdd6ba640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd80b08d0 con 0x7fecd80a98f0 2024-03-25T18:05:44.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.189+0000 7fecdd6ba640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd80b0b20 con 0x7fecd813f710 2024-03-25T18:05:44.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.189+0000 7fecdd6ba640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd80afb70 con 0x7fecd81430c0 2024-03-25T18:05:44.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.189+0000 7fecdd6ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:44.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.189+0000 7fecdd6ba640 10 monclient(hunting): authenticate will time out at 7472.768348s 2024-03-25T18:05:44.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.201+0000 7fecd6ffd640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fecd80a98f0 0x7fecd80af180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:52930/0 (socket says 172.21.15.18:52930) 2024-03-25T18:05:44.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.201+0000 7fecd6ffd640 1 -- 172.21.15.18:0/3173325874 learned_addr learned my addr 172.21.15.18:0/3173325874 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:05:44.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.201+0000 7fecbf7fe640 10 client.?.objecter ms_handle_connect 0x7fecd80a98f0 2024-03-25T18:05:44.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.201+0000 7fecbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:44.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.204+0000 7fecbf7fe640 10 client.?.objecter ms_handle_connect 0x7fecd81430c0 2024-03-25T18:05:44.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.204+0000 7fecbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:44.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.213+0000 7fecbf7fe640 10 client.?.objecter ms_handle_connect 0x7fecd813f710 2024-03-25T18:05:44.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.213+0000 7fecbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:44.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.214+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423438724 0 0) 0x7fecd80afb70 con 0x7fecd81430c0 2024-03-25T18:05:44.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.214+0000 7fecbf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:44.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.214+0000 7fecbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:44.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.214+0000 7fecbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:44.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.214+0000 7fecbf7fe640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:05:44.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.214+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc4004eb0 con 0x7fecd81430c0 2024-03-25T18:05:44.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.217+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562577008 0 0) 0x7fecd80b08d0 con 0x7fecd80a98f0 2024-03-25T18:05:44.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.217+0000 7fecbf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:44.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.217+0000 7fecbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:44.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.217+0000 7fecbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:44.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.217+0000 7fecbf7fe640 10 monclient(hunting): my global_id is 4647 2024-03-25T18:05:44.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.217+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecd80afb70 con 0x7fecd80a98f0 2024-03-25T18:05:44.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.224+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835456910 0 0) 0x7fecd80b0b20 con 0x7fecd813f710 2024-03-25T18:05:44.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.224+0000 7fecbf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:44.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.224+0000 7fecbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:44.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.224+0000 7fecbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:44.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.224+0000 7fecbf7fe640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:05:44.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.224+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecd80b08d0 con 0x7fecd813f710 2024-03-25T18:05:44.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.232+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2753542021 0 0) 0x7fecd80afb70 con 0x7fecd80a98f0 2024-03-25T18:05:44.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.232+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecd80b0b20 con 0x7fecd80a98f0 2024-03-25T18:05:44.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.232+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fecc8001ef0 con 0x7fecd80a98f0 2024-03-25T18:05:44.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.232+0000 7fecbf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:44.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.232+0000 7fecbf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:44.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.232+0000 7fecbf7fe640 10 monclient(hunting): dump: 2024-03-25T18:05:44.234 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:44.234 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:44.234 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:44.234 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:44.234 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:44.235 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:44.235 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:44.235 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:44.235 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:44.235 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:44.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.232+0000 7fecbf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:44.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.243+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4204090275 0 0) 0x7fecd80b08d0 con 0x7fecd813f710 2024-03-25T18:05:44.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.243+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecd80afb70 con 0x7fecd813f710 2024-03-25T18:05:44.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.243+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7feccc0017d0 con 0x7fecd813f710 2024-03-25T18:05:44.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.243+0000 7fecbf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:44.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.243+0000 7fecbf7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:44.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.243+0000 7fecbf7fe640 10 monclient(hunting): dump: 2024-03-25T18:05:44.245 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:44.245 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:44.245 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:44.245 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:44.245 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:44.245 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:44.246 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:44.246 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:44.246 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:44.246 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:44.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 354853057 0 0) 0x7fecd80b0b20 con 0x7fecd80a98f0 2024-03-25T18:05:44.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 >> v1:172.21.15.18:6790/0 conn(0x7fecd81430c0 legacy=0x7fecd8138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:44.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 >> v1:172.21.15.169:6789/0 conn(0x7fecd813f710 legacy=0x7fecd81d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:44.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecbf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:05:44.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecbf7fe640 1 monclient: found mon.a 2024-03-25T18:05:44.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:44.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecbf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:44.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd814e800 con 0x7fecd80a98f0 2024-03-25T18:05:44.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecbf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:05:44.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecbf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:05:44.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:44.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecdd6ba640 5 monclient: authenticate success, global_id 4647 2024-03-25T18:05:44.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecdd6ba640 10 monclient: _renew_subs 2024-03-25T18:05:44.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecdd6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:44.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecdd6ba640 1 -- 172.21.15.18:0/3173325874 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd81d7e60 con 0x7fecd80a98f0 2024-03-25T18:05:44.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecdd6ba640 10 client.4647.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:44.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecdd6ba640 10 monclient: _renew_subs 2024-03-25T18:05:44.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecdd6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:44.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecdd6ba640 1 -- 172.21.15.18:0/3173325874 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd8139850 con 0x7fecd80a98f0 2024-03-25T18:05:44.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.245+0000 7fecdd6ba640 1 librados: init done 2024-03-25T18:05:44.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.246+0000 7fecdd6ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:44.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.246+0000 7fecdd6ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:44.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.246+0000 7fecdd6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:44.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.246+0000 7fecdd6ba640 1 -- 172.21.15.18:0/3173325874 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd80af890 con 0x7fecd80a98f0 2024-03-25T18:05:44.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.553+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecc80020f0 con 0x7fecd80a98f0 2024-03-25T18:05:44.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.553+0000 7fecbf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:44.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.553+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fecc8004f70 con 0x7fecd80a98f0 2024-03-25T18:05:44.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.553+0000 7fecbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:44.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.553+0000 7fecbf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:44.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.553+0000 7fecbf7fe640 10 monclient: dump: 2024-03-25T18:05:44.555 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:44.555 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:44.555 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:44.555 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:44.555 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:44.555 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:44.555 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:44.555 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:44.556 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:44.556 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:44.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.553+0000 7fecd57fa640 4 set_mon_vals no callback set 2024-03-25T18:05:44.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.554+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fecc801af40 con 0x7fecd80a98f0 2024-03-25T18:05:44.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.554+0000 7fecd67fc640 10 monclient: get_auth_request con 0x7fecc406df80 auth_method 0 2024-03-25T18:05:44.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.555+0000 7fecbf7fe640 10 client.4647.objecter ms_handle_connect 0x7fecc406df80 2024-03-25T18:05:44.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.561+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fecc8054b40 con 0x7fecd80a98f0 2024-03-25T18:05:44.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.561+0000 7fecbf7fe640 10 client.4647.objecter ms_dispatch 0x7fecd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:44.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.561+0000 7fecbf7fe640 3 client.4647.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:44.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.561+0000 7fecbf7fe640 3 client.4647.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:44.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.561+0000 7fecbf7fe640 20 client.4647.objecter dump_active .. 0 homeless 2024-03-25T18:05:44.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.562+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fecc8050650 con 0x7fecd80a98f0 2024-03-25T18:05:44.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.562+0000 7fecbf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:44.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.562+0000 7fecbf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:44.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.818+0000 7fecdd6ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:44.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.818+0000 7fecdd6ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:44.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.818+0000 7fecdd6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:44.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.818+0000 7fecdd6ba640 1 -- 172.21.15.18:0/3173325874 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fecd814e990 con 0x7fecd80a98f0 2024-03-25T18:05:44.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.851+0000 7fecbf7fe640 1 -- 172.21.15.18:0/3173325874 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fecd814e990 con 0x7fecd80a98f0 2024-03-25T18:05:44.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.851+0000 7fecbf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:44.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.851+0000 7fecbf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:44.853 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:05:44.853 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:44.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.854+0000 7fecdd6ba640 10 librados: watch_flush enter 2024-03-25T18:05:44.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.854+0000 7fecdd6ba640 10 librados: watch_flush exit 2024-03-25T18:05:44.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.854+0000 7fecdd6ba640 20 client.4647.objecter shutdown clearing up homeless session... 2024-03-25T18:05:44.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.854+0000 7fecdd6ba640 10 client.4647.objecter successfully canceled tick 2024-03-25T18:05:44.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.854+0000 7fecdd6ba640 1 -- 172.21.15.18:0/3173325874 >> v1:172.21.15.18:6800/3721244451 conn(0x7fecc406df80 legacy=0x7fecc4070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:44.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.854+0000 7fecdd6ba640 10 monclient: shutdown 2024-03-25T18:05:44.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.854+0000 7fecdd6ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:44.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.854+0000 7fecdd6ba640 1 -- 172.21.15.18:0/3173325874 >> v1:172.21.15.18:6789/0 conn(0x7fecd80a98f0 legacy=0x7fecd80af180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:44.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.854+0000 7fecdd6ba640 1 -- 172.21.15.18:0/3173325874 shutdown_connections 2024-03-25T18:05:44.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.854+0000 7fecdd6ba640 1 -- 172.21.15.18:0/3173325874 >> 172.21.15.18:0/3173325874 conn(0x7fecd80b7ea0 msgr2=0x7fecd8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:44.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.854+0000 7fecdd6ba640 1 -- 172.21.15.18:0/3173325874 shutdown_connections 2024-03-25T18:05:44.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.854+0000 7fecdd6ba640 1 -- 172.21.15.18:0/3173325874 wait complete. 2024-03-25T18:05:44.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:44.855+0000 7fecdd6ba640 1 librados: shutdown 2024-03-25T18:05:44.867 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:48.954 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:46.746 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:48.964 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:46.713 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:48.967 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:46.760 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:48.996 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:09:46.789 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:49.025 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:46.774 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:49.034 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:46.827 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:49.039 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:46.831 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:49.056 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:09:46.804 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:49.061 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:09:46.853 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:51.673 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:51.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.788+0000 7f07435e6640 10 monclient: get_monmap_and_config 2024-03-25T18:05:51.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.788+0000 7f07435e6640 10 monclient: build_initial_monmap 2024-03-25T18:05:51.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.788+0000 7f07435e6640 10 monclient: monmap: 2024-03-25T18:05:51.790 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:05:51.790 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:51.790 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:05:51.790188+0000 2024-03-25T18:05:51.790 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:05:51.790188+0000 2024-03-25T18:05:51.790 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:51.790 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:51.790 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:05:51.790 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:05:51.790 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:05:51.790 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:51.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.789+0000 7f07435e6640 1 Processor -- start 2024-03-25T18:05:51.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.789+0000 7f07435e6640 1 -- start start 2024-03-25T18:05:51.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.789+0000 7f07435e6640 10 monclient: init 2024-03-25T18:05:51.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.790+0000 7f07435e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:51.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.790+0000 7f07435e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:05:51.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.790+0000 7f07435e6640 10 monclient(hunting): picked mon.noname-a con 0x7f073c0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:05:51.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.790+0000 7f07435e6640 10 monclient(hunting): picked mon.noname-b con 0x7f073c13f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:05:51.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.790+0000 7f07435e6640 10 monclient(hunting): picked mon.noname-c con 0x7f073c1430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:05:51.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.790+0000 7f07435e6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073c142520 con 0x7f073c0a98f0 2024-03-25T18:05:51.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.790+0000 7f07435e6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073c0b28a0 con 0x7f073c1430c0 2024-03-25T18:05:51.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.790+0000 7f07435e6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073c0b2a30 con 0x7f073c13f710 2024-03-25T18:05:51.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.790+0000 7f07435e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:51.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.790+0000 7f07435e6640 10 monclient(hunting): authenticate will time out at 7480.369379s 2024-03-25T18:05:51.798 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.797+0000 7f0740b81640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f073c13f710 0x7f073c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:54474/0 (socket says 172.21.15.18:54474) 2024-03-25T18:05:51.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.797+0000 7f0740b81640 1 -- 172.21.15.18:0/2696047919 learned_addr learned my addr 172.21.15.18:0/2696047919 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:05:51.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.807+0000 7f0733fff640 1 -- 172.21.15.18:0/2696047919 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351592934 0 0) 0x7f073c0b2a30 con 0x7f073c13f710 2024-03-25T18:05:51.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.807+0000 7f0733fff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:51.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.807+0000 7f0733fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:51.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.807+0000 7f0733fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:51.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.807+0000 7f0733fff640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:05:51.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.807+0000 7f0733fff640 1 -- 172.21.15.18:0/2696047919 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0728003520 con 0x7f073c13f710 2024-03-25T18:05:51.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.813+0000 7f0733fff640 1 -- 172.21.15.18:0/2696047919 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2328818414 0 0) 0x7f0728003520 con 0x7f073c13f710 2024-03-25T18:05:51.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.813+0000 7f0733fff640 1 -- 172.21.15.18:0/2696047919 >> v1:172.21.15.169:6789/0 conn(0x7f073c1430c0 legacy=0x7f073c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:51.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.813+0000 7f0733fff640 1 -- 172.21.15.18:0/2696047919 >> v1:172.21.15.18:6789/0 conn(0x7f073c0a98f0 legacy=0x7f073c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:51.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.813+0000 7f0733fff640 10 monclient: _finish_hunting 0 2024-03-25T18:05:51.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.813+0000 7f0733fff640 1 monclient: found mon.noname-b 2024-03-25T18:05:51.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.813+0000 7f0733fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:51.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.813+0000 7f0733fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:05:51.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.813+0000 7f0733fff640 1 -- 172.21.15.18:0/2696047919 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073c0b44c0 con 0x7f073c13f710 2024-03-25T18:05:51.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.813+0000 7f0733fff640 10 monclient: _finish_auth 0 2024-03-25T18:05:51.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.813+0000 7f0733fff640 10 monclient: _check_auth_tickets 2024-03-25T18:05:51.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.813+0000 7f0733fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:51.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.813+0000 7f07435e6640 5 monclient: authenticate success, global_id 4658 2024-03-25T18:05:51.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.813+0000 7f07435e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:51.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.825+0000 7f0733fff640 1 -- 172.21.15.18:0/2696047919 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f072c001690 con 0x7f073c13f710 2024-03-25T18:05:51.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.825+0000 7f0733fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:51.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.825+0000 7f0733fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:51.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.825+0000 7f0733fff640 10 monclient: dump: 2024-03-25T18:05:51.827 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:51.827 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:51.827 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:51.827 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:51.827 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:51.827 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:51.827 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:51.827 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:51.827 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:51.827 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:51.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.825+0000 7f0733fff640 1 -- 172.21.15.18:0/2696047919 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f072c001920 con 0x7f073c13f710 2024-03-25T18:05:51.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.825+0000 7f07435e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:51.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.825+0000 7f0733fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:51.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.825+0000 7f0733fff640 1 -- 172.21.15.18:0/2696047919 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f072c001c30 con 0x7f073c13f710 2024-03-25T18:05:51.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.825+0000 7f07435e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:51.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.825+0000 7f07435e6640 4 set_mon_vals no callback set 2024-03-25T18:05:51.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.825+0000 7f0733fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:51.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.825+0000 7f0733fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:51.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.825+0000 7f0733fff640 10 monclient: dump: 2024-03-25T18:05:51.828 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:51.828 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:51.828 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:51.828 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:51.828 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:51.828 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:51.828 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:51.828 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:51.829 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:51.829 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:51.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.826+0000 7f07435e6640 10 monclient: shutdown 2024-03-25T18:05:51.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.826+0000 7f07435e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:51.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.826+0000 7f07435e6640 1 -- 172.21.15.18:0/2696047919 >> v1:172.21.15.18:6790/0 conn(0x7f073c13f710 legacy=0x7f073c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:51.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.826+0000 7f07435e6640 1 -- 172.21.15.18:0/2696047919 shutdown_connections 2024-03-25T18:05:51.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.826+0000 7f07435e6640 1 -- 172.21.15.18:0/2696047919 >> 172.21.15.18:0/2696047919 conn(0x7f073c0b7ea0 msgr2=0x7f073c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:51.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.826+0000 7f07435e6640 1 -- 172.21.15.18:0/2696047919 shutdown_connections 2024-03-25T18:05:51.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.826+0000 7f07435e6640 1 -- 172.21.15.18:0/2696047919 wait complete. 2024-03-25T18:05:51.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.827+0000 7f07435e6640 10 monclient: build_initial_monmap 2024-03-25T18:05:51.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.827+0000 7f07435e6640 10 monclient: monmap: 2024-03-25T18:05:51.831 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:05:51.831 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:51.831 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:05:51.831 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:05:51.831 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:51.831 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:05:51.832 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:05:51.832 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:05:51.832 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:05:51.832 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:51.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.827+0000 7f07435e6640 1 librados: starting msgr at 2024-03-25T18:05:51.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.827+0000 7f07435e6640 1 librados: starting objecter 2024-03-25T18:05:51.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.827+0000 7f07435e6640 1 Processor -- start 2024-03-25T18:05:51.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.827+0000 7f07435e6640 1 -- start start 2024-03-25T18:05:51.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.827+0000 7f07435e6640 1 librados: setting wanted keys 2024-03-25T18:05:51.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.827+0000 7f07435e6640 1 librados: calling monclient init 2024-03-25T18:05:51.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.827+0000 7f07435e6640 10 monclient: init 2024-03-25T18:05:51.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.828+0000 7f07435e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:51.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.828+0000 7f07435e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:05:51.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.828+0000 7f07435e6640 10 monclient(hunting): picked mon.noname-b con 0x7f073c0a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:05:51.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.828+0000 7f07435e6640 10 monclient(hunting): picked mon.noname-c con 0x7f073c13f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:05:51.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.828+0000 7f07435e6640 10 monclient(hunting): picked mon.noname-a con 0x7f073c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:05:51.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.828+0000 7f07435e6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073c137130 con 0x7f073c1430c0 2024-03-25T18:05:51.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.828+0000 7f07435e6640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073c137360 con 0x7f073c0a98f0 2024-03-25T18:05:51.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.828+0000 7f07435e6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073c1381e0 con 0x7f073c13f710 2024-03-25T18:05:51.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.828+0000 7f07435e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:51.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.828+0000 7f07435e6640 10 monclient(hunting): authenticate will time out at 7480.407240s 2024-03-25T18:05:51.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.835+0000 7f0740b81640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f073c13f710 0x7f073c137ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:54498/0 (socket says 172.21.15.18:54498) 2024-03-25T18:05:51.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.835+0000 7f0740b81640 1 -- 172.21.15.18:0/3306812045 learned_addr learned my addr 172.21.15.18:0/3306812045 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:05:51.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.839+0000 7f0731ffb640 10 client.?.objecter ms_handle_connect 0x7f073c13f710 2024-03-25T18:05:51.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.839+0000 7f0731ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:51.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.844+0000 7f0731ffb640 10 client.?.objecter ms_handle_connect 0x7f073c0a98f0 2024-03-25T18:05:51.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.844+0000 7f0731ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:51.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.848+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022092527 0 0) 0x7f073c1381e0 con 0x7f073c13f710 2024-03-25T18:05:51.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.848+0000 7f0731ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:51.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.848+0000 7f0731ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:51.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.848+0000 7f0731ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:51.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.848+0000 7f0731ffb640 10 monclient(hunting): my global_id is 4661 2024-03-25T18:05:51.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.848+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0720004eb0 con 0x7f073c13f710 2024-03-25T18:05:51.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.859+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864964980 0 0) 0x7f073c137360 con 0x7f073c0a98f0 2024-03-25T18:05:51.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.859+0000 7f0731ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:51.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.859+0000 7f0731ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:51.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.859+0000 7f0731ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:51.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.859+0000 7f0731ffb640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:05:51.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.859+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f073c1381e0 con 0x7f073c0a98f0 2024-03-25T18:05:51.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.859+0000 7f0731ffb640 10 client.?.objecter ms_handle_connect 0x7f073c1430c0 2024-03-25T18:05:51.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.859+0000 7f0731ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:51.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.869+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420777114 0 0) 0x7f073c137130 con 0x7f073c1430c0 2024-03-25T18:05:51.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.869+0000 7f0731ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:51.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.869+0000 7f0731ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:51.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.869+0000 7f0731ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:51.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.869+0000 7f0731ffb640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:05:51.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.869+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f073c137360 con 0x7f073c1430c0 2024-03-25T18:05:51.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.871+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2860340664 0 0) 0x7f0720004eb0 con 0x7f073c13f710 2024-03-25T18:05:51.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.871+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f073c137130 con 0x7f073c13f710 2024-03-25T18:05:51.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.871+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f072c001650 con 0x7f073c13f710 2024-03-25T18:05:51.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.871+0000 7f0731ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:51.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.871+0000 7f0731ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:51.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.871+0000 7f0731ffb640 10 monclient(hunting): dump: 2024-03-25T18:05:51.873 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:51.873 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:51.873 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:51.873 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:51.873 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:51.873 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:51.873 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:51.873 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:51.873 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:51.873 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:51.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.871+0000 7f0731ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:51.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.880+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2717699698 0 0) 0x7f073c1381e0 con 0x7f073c0a98f0 2024-03-25T18:05:51.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.880+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0720004eb0 con 0x7f073c0a98f0 2024-03-25T18:05:51.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.880+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0738001930 con 0x7f073c0a98f0 2024-03-25T18:05:51.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.880+0000 7f0731ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:51.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.880+0000 7f0731ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:51.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.880+0000 7f0731ffb640 10 monclient(hunting): dump: 2024-03-25T18:05:51.882 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:51.882 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:51.882 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:51.882 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:51.882 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:51.882 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:51.882 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:51.882 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:51.882 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:51.882 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:51.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.886+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2648410668 0 0) 0x7f0720004eb0 con 0x7f073c0a98f0 2024-03-25T18:05:51.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.886+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 >> v1:172.21.15.18:6790/0 conn(0x7f073c13f710 legacy=0x7f073c137ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:51.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.886+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 >> v1:172.21.15.18:6789/0 conn(0x7f073c1430c0 legacy=0x7f073c1e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:51.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.886+0000 7f0731ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:05:51.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.886+0000 7f0731ffb640 1 monclient: found mon.b 2024-03-25T18:05:51.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.886+0000 7f0731ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:51.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.886+0000 7f0731ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:05:51.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.886+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073c135100 con 0x7f073c0a98f0 2024-03-25T18:05:51.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.886+0000 7f0731ffb640 10 monclient: _finish_auth 0 2024-03-25T18:05:51.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.886+0000 7f0731ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:05:51.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.886+0000 7f0731ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:51.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.886+0000 7f07435e6640 5 monclient: authenticate success, global_id 4642 2024-03-25T18:05:51.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.886+0000 7f07435e6640 10 monclient: _renew_subs 2024-03-25T18:05:51.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.886+0000 7f07435e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:05:51.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.886+0000 7f07435e6640 1 -- 172.21.15.18:0/3306812045 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f073c0b6f60 con 0x7f073c0a98f0 2024-03-25T18:05:51.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.887+0000 7f07435e6640 10 client.4642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:51.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.887+0000 7f07435e6640 10 monclient: _renew_subs 2024-03-25T18:05:51.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.887+0000 7f07435e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:05:51.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.887+0000 7f07435e6640 1 -- 172.21.15.18:0/3306812045 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f073c136420 con 0x7f073c0a98f0 2024-03-25T18:05:51.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.887+0000 7f07435e6640 1 librados: init done 2024-03-25T18:05:51.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.887+0000 7f07435e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:51.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.887+0000 7f07435e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:51.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.887+0000 7f07435e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:05:51.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:51.887+0000 7f07435e6640 1 -- 172.21.15.18:0/3306812045 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f073c0627b0 con 0x7f073c0a98f0 2024-03-25T18:05:52.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.162+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0738003610 con 0x7f073c0a98f0 2024-03-25T18:05:52.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.162+0000 7f0731ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:52.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.162+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0738003920 con 0x7f073c0a98f0 2024-03-25T18:05:52.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.162+0000 7f0731ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:52.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.162+0000 7f0731ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:52.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.162+0000 7f0731ffb640 10 monclient: dump: 2024-03-25T18:05:52.164 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:52.164 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:52.164 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:52.164 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:52.165 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:52.165 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:52.165 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:52.165 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:52.165 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:52.165 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:52.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.162+0000 7f07337fe640 4 set_mon_vals no callback set 2024-03-25T18:05:52.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.163+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f07380198f0 con 0x7f073c0a98f0 2024-03-25T18:05:52.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.164+0000 7f0740b81640 10 monclient: get_auth_request con 0x7f072006df20 auth_method 0 2024-03-25T18:05:52.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.164+0000 7f0731ffb640 10 client.4642.objecter ms_handle_connect 0x7f072006df20 2024-03-25T18:05:52.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.168+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f0738053620 con 0x7f073c0a98f0 2024-03-25T18:05:52.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.168+0000 7f0731ffb640 10 client.4642.objecter ms_dispatch 0x7f073c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:52.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.168+0000 7f0731ffb640 3 client.4642.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:52.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.168+0000 7f0731ffb640 3 client.4642.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:52.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.168+0000 7f0731ffb640 20 client.4642.objecter dump_active .. 0 homeless 2024-03-25T18:05:52.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.169+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f073804ef60 con 0x7f073c0a98f0 2024-03-25T18:05:52.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.169+0000 7f0731ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:52.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.169+0000 7f0731ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:52.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.407+0000 7f07435e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:52.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.407+0000 7f07435e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:52.408 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.407+0000 7f07435e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:05:52.409 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.407+0000 7f07435e6640 1 -- 172.21.15.18:0/3306812045 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f073c13e230 con 0x7f073c0a98f0 2024-03-25T18:05:52.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.443+0000 7f0731ffb640 1 -- 172.21.15.18:0/3306812045 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f0738051060 con 0x7f073c0a98f0 2024-03-25T18:05:52.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.443+0000 7f0731ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:52.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.443+0000 7f0731ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:52.444 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:05:52.445 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:52.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.445+0000 7f07435e6640 10 librados: watch_flush enter 2024-03-25T18:05:52.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.445+0000 7f07435e6640 10 librados: watch_flush exit 2024-03-25T18:05:52.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.445+0000 7f07435e6640 20 client.4642.objecter shutdown clearing up homeless session... 2024-03-25T18:05:52.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.445+0000 7f07435e6640 10 client.4642.objecter successfully canceled tick 2024-03-25T18:05:52.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.446+0000 7f07435e6640 1 -- 172.21.15.18:0/3306812045 >> v1:172.21.15.18:6800/3721244451 conn(0x7f072006df20 legacy=0x7f07200703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:52.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.446+0000 7f07435e6640 10 monclient: shutdown 2024-03-25T18:05:52.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.446+0000 7f07435e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:52.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.446+0000 7f07435e6640 1 -- 172.21.15.18:0/3306812045 >> v1:172.21.15.169:6789/0 conn(0x7f073c0a98f0 legacy=0x7f073c135850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:52.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.446+0000 7f07435e6640 1 -- 172.21.15.18:0/3306812045 shutdown_connections 2024-03-25T18:05:52.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.446+0000 7f07435e6640 1 -- 172.21.15.18:0/3306812045 >> 172.21.15.18:0/3306812045 conn(0x7f073c0b7ea0 msgr2=0x7f073c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:52.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.446+0000 7f07435e6640 1 -- 172.21.15.18:0/3306812045 shutdown_connections 2024-03-25T18:05:52.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.446+0000 7f07435e6640 1 -- 172.21.15.18:0/3306812045 wait complete. 2024-03-25T18:05:52.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:52.446+0000 7f07435e6640 1 librados: shutdown 2024-03-25T18:05:52.458 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:59.265 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:59.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.381+0000 7fa7e3315640 10 monclient: get_monmap_and_config 2024-03-25T18:05:59.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.381+0000 7fa7e3315640 10 monclient: build_initial_monmap 2024-03-25T18:05:59.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.381+0000 7fa7e3315640 10 monclient: monmap: 2024-03-25T18:05:59.383 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:05:59.383 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:59.383 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:05:59.383220+0000 2024-03-25T18:05:59.383 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:05:59.383220+0000 2024-03-25T18:05:59.383 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:59.383 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:59.383 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:05:59.384 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:05:59.384 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:05:59.384 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:59.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.382+0000 7fa7e3315640 1 Processor -- start 2024-03-25T18:05:59.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.382+0000 7fa7e3315640 1 -- start start 2024-03-25T18:05:59.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.382+0000 7fa7e3315640 10 monclient: init 2024-03-25T18:05:59.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.383+0000 7fa7e3315640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:59.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.383+0000 7fa7e3315640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:05:59.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.383+0000 7fa7e3315640 10 monclient(hunting): picked mon.noname-c con 0x7fa7dc0a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:05:59.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.383+0000 7fa7e3315640 10 monclient(hunting): picked mon.noname-b con 0x7fa7dc13f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:05:59.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.383+0000 7fa7e3315640 10 monclient(hunting): picked mon.noname-a con 0x7fa7dc1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:05:59.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.383+0000 7fa7e3315640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7dc142520 con 0x7fa7dc1430c0 2024-03-25T18:05:59.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.383+0000 7fa7e3315640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7dc0b28a0 con 0x7fa7dc0a98f0 2024-03-25T18:05:59.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.383+0000 7fa7e3315640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7dc0b2a30 con 0x7fa7dc13f710 2024-03-25T18:05:59.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.383+0000 7fa7e3315640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:59.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.383+0000 7fa7e3315640 10 monclient(hunting): authenticate will time out at 7487.962039s 2024-03-25T18:05:59.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.386+0000 7fa7e08b0640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fa7dc13f710 0x7fa7dc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:58208/0 (socket says 172.21.15.18:58208) 2024-03-25T18:05:59.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.387+0000 7fa7e08b0640 1 -- 172.21.15.18:0/2905029459 learned_addr learned my addr 172.21.15.18:0/2905029459 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:05:59.404 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.402+0000 7fa7d3fff640 1 -- 172.21.15.18:0/2905029459 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20949440 0 0) 0x7fa7dc0b2a30 con 0x7fa7dc13f710 2024-03-25T18:05:59.404 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.402+0000 7fa7d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:59.404 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.402+0000 7fa7d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:59.404 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.402+0000 7fa7d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:59.404 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.402+0000 7fa7d3fff640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:05:59.404 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.402+0000 7fa7d3fff640 1 -- 172.21.15.18:0/2905029459 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7c8003520 con 0x7fa7dc13f710 2024-03-25T18:05:59.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7d3fff640 1 -- 172.21.15.18:0/2905029459 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2280162366 0 0) 0x7fa7c8003520 con 0x7fa7dc13f710 2024-03-25T18:05:59.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7d3fff640 1 -- 172.21.15.18:0/2905029459 >> v1:172.21.15.169:6789/0 conn(0x7fa7dc0a98f0 legacy=0x7fa7dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:59.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7d3fff640 1 -- 172.21.15.18:0/2905029459 >> v1:172.21.15.18:6789/0 conn(0x7fa7dc1430c0 legacy=0x7fa7dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:59.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7d3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:05:59.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7d3fff640 1 monclient: found mon.noname-b 2024-03-25T18:05:59.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:59.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7d3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:05:59.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7d3fff640 1 -- 172.21.15.18:0/2905029459 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7dc0b44c0 con 0x7fa7dc13f710 2024-03-25T18:05:59.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7d3fff640 10 monclient: _finish_auth 0 2024-03-25T18:05:59.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7d3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:05:59.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:59.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7e3315640 5 monclient: authenticate success, global_id 4664 2024-03-25T18:05:59.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7d3fff640 1 -- 172.21.15.18:0/2905029459 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa7cc001690 con 0x7fa7dc13f710 2024-03-25T18:05:59.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7e3315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:59.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:59.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:59.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7d3fff640 10 monclient: dump: 2024-03-25T18:05:59.426 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:59.426 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:59.426 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:59.426 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:59.426 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:59.426 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:59.426 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:59.427 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:59.427 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:59.427 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:59.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.423+0000 7fa7e3315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:59.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.429+0000 7fa7d3fff640 1 -- 172.21.15.18:0/2905029459 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7cc0017f0 con 0x7fa7dc13f710 2024-03-25T18:05:59.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.429+0000 7fa7d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:59.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.429+0000 7fa7e3315640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:59.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.429+0000 7fa7e3315640 4 set_mon_vals no callback set 2024-03-25T18:05:59.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.429+0000 7fa7d3fff640 1 -- 172.21.15.18:0/2905029459 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa7cc001c00 con 0x7fa7dc13f710 2024-03-25T18:05:59.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.429+0000 7fa7d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:59.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.429+0000 7fa7d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:59.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.429+0000 7fa7d3fff640 10 monclient: dump: 2024-03-25T18:05:59.431 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:59.431 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:59.431 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:59.431 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:59.431 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:59.431 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:59.431 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:59.431 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:59.432 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:59.432 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:59.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.430+0000 7fa7e3315640 10 monclient: shutdown 2024-03-25T18:05:59.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.430+0000 7fa7e3315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:59.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.430+0000 7fa7e3315640 1 -- 172.21.15.18:0/2905029459 >> v1:172.21.15.18:6790/0 conn(0x7fa7dc13f710 legacy=0x7fa7dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:59.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.430+0000 7fa7e3315640 1 -- 172.21.15.18:0/2905029459 shutdown_connections 2024-03-25T18:05:59.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.430+0000 7fa7e3315640 1 -- 172.21.15.18:0/2905029459 >> 172.21.15.18:0/2905029459 conn(0x7fa7dc0b7ea0 msgr2=0x7fa7dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:59.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.430+0000 7fa7e3315640 1 -- 172.21.15.18:0/2905029459 shutdown_connections 2024-03-25T18:05:59.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.430+0000 7fa7e3315640 1 -- 172.21.15.18:0/2905029459 wait complete. 2024-03-25T18:05:59.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.431+0000 7fa7e3315640 10 monclient: build_initial_monmap 2024-03-25T18:05:59.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.431+0000 7fa7e3315640 10 monclient: monmap: 2024-03-25T18:05:59.434 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:05:59.434 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:59.435 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:05:59.435 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:05:59.435 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:59.435 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:05:59.435 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:05:59.435 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:05:59.435 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:05:59.435 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:59.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.431+0000 7fa7e3315640 1 librados: starting msgr at 2024-03-25T18:05:59.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.431+0000 7fa7e3315640 1 librados: starting objecter 2024-03-25T18:05:59.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.431+0000 7fa7e3315640 1 Processor -- start 2024-03-25T18:05:59.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.431+0000 7fa7e3315640 1 -- start start 2024-03-25T18:05:59.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.431+0000 7fa7e3315640 1 librados: setting wanted keys 2024-03-25T18:05:59.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.431+0000 7fa7e3315640 1 librados: calling monclient init 2024-03-25T18:05:59.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.431+0000 7fa7e3315640 10 monclient: init 2024-03-25T18:05:59.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.432+0000 7fa7e3315640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:59.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.432+0000 7fa7e3315640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:05:59.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.432+0000 7fa7e3315640 10 monclient(hunting): picked mon.noname-c con 0x7fa7dc0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:05:59.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.432+0000 7fa7e3315640 10 monclient(hunting): picked mon.noname-b con 0x7fa7dc13f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:05:59.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.432+0000 7fa7e3315640 10 monclient(hunting): picked mon.noname-a con 0x7fa7dc1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:05:59.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.432+0000 7fa7e3315640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7dc1daee0 con 0x7fa7dc1430c0 2024-03-25T18:05:59.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.432+0000 7fa7e3315640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7dc1d51f0 con 0x7fa7dc13f710 2024-03-25T18:05:59.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.432+0000 7fa7e3315640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7dc138aa0 con 0x7fa7dc0a98f0 2024-03-25T18:05:59.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.432+0000 7fa7e3315640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:59.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.432+0000 7fa7e3315640 10 monclient(hunting): authenticate will time out at 7488.011230s 2024-03-25T18:05:59.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.443+0000 7fa7e18b2640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fa7dc1430c0 0x7fa7dc1e1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:51606/0 (socket says 172.21.15.18:51606) 2024-03-25T18:05:59.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.443+0000 7fa7e18b2640 1 -- 172.21.15.18:0/2237891702 learned_addr learned my addr 172.21.15.18:0/2237891702 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:05:59.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.444+0000 7fa7d1ffb640 10 client.?.objecter ms_handle_connect 0x7fa7dc1430c0 2024-03-25T18:05:59.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.444+0000 7fa7d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:59.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.452+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571327077 0 0) 0x7fa7dc1daee0 con 0x7fa7dc1430c0 2024-03-25T18:05:59.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.452+0000 7fa7d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:59.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.452+0000 7fa7d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:59.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.452+0000 7fa7d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:59.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.452+0000 7fa7d1ffb640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:05:59.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.452+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7c0004eb0 con 0x7fa7dc1430c0 2024-03-25T18:05:59.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.455+0000 7fa7d1ffb640 10 client.?.objecter ms_handle_connect 0x7fa7dc0a98f0 2024-03-25T18:05:59.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.455+0000 7fa7d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:59.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.458+0000 7fa7d1ffb640 10 client.?.objecter ms_handle_connect 0x7fa7dc13f710 2024-03-25T18:05:59.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.458+0000 7fa7d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:59.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.466+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1732552156 0 0) 0x7fa7c0004eb0 con 0x7fa7dc1430c0 2024-03-25T18:05:59.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.466+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7dc1daee0 con 0x7fa7dc1430c0 2024-03-25T18:05:59.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.466+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa7d4001c90 con 0x7fa7dc1430c0 2024-03-25T18:05:59.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.466+0000 7fa7d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:59.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.466+0000 7fa7d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:59.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.466+0000 7fa7d1ffb640 10 monclient(hunting): dump: 2024-03-25T18:05:59.468 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:59.468 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:59.468 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:59.468 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:59.468 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:59.468 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:59.468 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:59.468 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:59.468 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:59.468 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:59.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.466+0000 7fa7d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:59.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.468+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521055552 0 0) 0x7fa7dc138aa0 con 0x7fa7dc0a98f0 2024-03-25T18:05:59.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.468+0000 7fa7d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:59.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.468+0000 7fa7d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:59.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.468+0000 7fa7d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:59.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.468+0000 7fa7d1ffb640 10 monclient(hunting): my global_id is 4667 2024-03-25T18:05:59.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.468+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7c0004eb0 con 0x7fa7dc0a98f0 2024-03-25T18:05:59.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.475+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769282295 0 0) 0x7fa7dc1d51f0 con 0x7fa7dc13f710 2024-03-25T18:05:59.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.475+0000 7fa7d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:59.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.475+0000 7fa7d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:59.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.475+0000 7fa7d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:59.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.475+0000 7fa7d1ffb640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:05:59.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.475+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7dc138aa0 con 0x7fa7dc13f710 2024-03-25T18:05:59.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3205897072 0 0) 0x7fa7dc1daee0 con 0x7fa7dc1430c0 2024-03-25T18:05:59.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 >> v1:172.21.15.18:6790/0 conn(0x7fa7dc0a98f0 legacy=0x7fa7dc1db400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:59.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 >> v1:172.21.15.169:6789/0 conn(0x7fa7dc13f710 legacy=0x7fa7dc138390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:59.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7d1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:05:59.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7d1ffb640 1 monclient: found mon.a 2024-03-25T18:05:59.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:59.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7d1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:59.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7dc1dab70 con 0x7fa7dc1430c0 2024-03-25T18:05:59.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7d1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:05:59.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7d1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:05:59.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:59.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7e3315640 5 monclient: authenticate success, global_id 4659 2024-03-25T18:05:59.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7e3315640 10 monclient: _renew_subs 2024-03-25T18:05:59.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7e3315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:59.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7e3315640 1 -- 172.21.15.18:0/2237891702 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7dc138d80 con 0x7fa7dc1430c0 2024-03-25T18:05:59.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7e3315640 10 client.4659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:59.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7e3315640 10 monclient: _renew_subs 2024-03-25T18:05:59.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7e3315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:59.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7e3315640 1 -- 172.21.15.18:0/2237891702 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7dc139120 con 0x7fa7dc1430c0 2024-03-25T18:05:59.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.478+0000 7fa7e3315640 1 librados: init done 2024-03-25T18:05:59.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.479+0000 7fa7e3315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:59.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.479+0000 7fa7e3315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:59.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.479+0000 7fa7e3315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:05:59.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.479+0000 7fa7e3315640 1 -- 172.21.15.18:0/2237891702 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7dc0627b0 con 0x7fa7dc1430c0 2024-03-25T18:05:59.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.805+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7d4003990 con 0x7fa7dc1430c0 2024-03-25T18:05:59.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.805+0000 7fa7d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:59.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.805+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa7d4003d20 con 0x7fa7dc1430c0 2024-03-25T18:05:59.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.805+0000 7fa7d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:59.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.805+0000 7fa7d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:59.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.805+0000 7fa7d1ffb640 10 monclient: dump: 2024-03-25T18:05:59.807 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:05:59.807 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:05:59.807 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:59.807 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:05:59.808 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:59.808 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:05:59.808 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:05:59.808 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:05:59.808 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:05:59.808 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:05:59.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.805+0000 7fa7d37fe640 4 set_mon_vals no callback set 2024-03-25T18:05:59.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.806+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fa7d4003f40 con 0x7fa7dc1430c0 2024-03-25T18:05:59.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.807+0000 7fa7e10b1640 10 monclient: get_auth_request con 0x7fa7c006de60 auth_method 0 2024-03-25T18:05:59.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.807+0000 7fa7d1ffb640 10 client.4659.objecter ms_handle_connect 0x7fa7c006de60 2024-03-25T18:05:59.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.813+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fa7d4054ac0 con 0x7fa7dc1430c0 2024-03-25T18:05:59.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.813+0000 7fa7d1ffb640 10 client.4659.objecter ms_dispatch 0x7fa7dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:59.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.813+0000 7fa7d1ffb640 3 client.4659.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:59.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.813+0000 7fa7d1ffb640 3 client.4659.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:59.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.813+0000 7fa7d1ffb640 20 client.4659.objecter dump_active .. 0 homeless 2024-03-25T18:05:59.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.813+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7d40505d0 con 0x7fa7dc1430c0 2024-03-25T18:05:59.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.813+0000 7fa7d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:59.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:05:59.813+0000 7fa7d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:00.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.067+0000 7fa7e3315640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:00.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.067+0000 7fa7e3315640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:00.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.067+0000 7fa7e3315640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:06:00.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.067+0000 7fa7e3315640 1 -- 172.21.15.18:0/2237891702 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7dc069f40 con 0x7fa7dc1430c0 2024-03-25T18:06:00.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.088+0000 7fa7d1ffb640 1 -- 172.21.15.18:0/2237891702 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fa7d4052500 con 0x7fa7dc1430c0 2024-03-25T18:06:00.090 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.088+0000 7fa7d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:00.090 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.088+0000 7fa7d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:00.090 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:06:00.090 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:00.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.090+0000 7fa7e3315640 10 librados: watch_flush enter 2024-03-25T18:06:00.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.090+0000 7fa7e3315640 10 librados: watch_flush exit 2024-03-25T18:06:00.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.090+0000 7fa7e3315640 20 client.4659.objecter shutdown clearing up homeless session... 2024-03-25T18:06:00.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.090+0000 7fa7e3315640 10 client.4659.objecter successfully canceled tick 2024-03-25T18:06:00.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.091+0000 7fa7e3315640 1 -- 172.21.15.18:0/2237891702 >> v1:172.21.15.18:6800/3721244451 conn(0x7fa7c006de60 legacy=0x7fa7c0070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:00.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.091+0000 7fa7e3315640 10 monclient: shutdown 2024-03-25T18:06:00.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.091+0000 7fa7e3315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:00.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.091+0000 7fa7e3315640 1 -- 172.21.15.18:0/2237891702 >> v1:172.21.15.18:6789/0 conn(0x7fa7dc1430c0 legacy=0x7fa7dc1e1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:00.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.091+0000 7fa7e3315640 1 -- 172.21.15.18:0/2237891702 shutdown_connections 2024-03-25T18:06:00.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.091+0000 7fa7e3315640 1 -- 172.21.15.18:0/2237891702 >> 172.21.15.18:0/2237891702 conn(0x7fa7dc0b7ea0 msgr2=0x7fa7dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:00.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.091+0000 7fa7e3315640 1 -- 172.21.15.18:0/2237891702 shutdown_connections 2024-03-25T18:06:00.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.091+0000 7fa7e3315640 1 -- 172.21.15.18:0/2237891702 wait complete. 2024-03-25T18:06:00.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:00.091+0000 7fa7e3315640 1 librados: shutdown 2024-03-25T18:06:00.104 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:06.910 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:07.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.025+0000 7fc769ff4640 10 monclient: get_monmap_and_config 2024-03-25T18:06:07.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.025+0000 7fc769ff4640 10 monclient: build_initial_monmap 2024-03-25T18:06:07.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.025+0000 7fc769ff4640 10 monclient: monmap: 2024-03-25T18:06:07.025 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:06:07.025 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:07.026 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:06:07.025554+0000 2024-03-25T18:06:07.026 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:06:07.025554+0000 2024-03-25T18:06:07.026 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:07.026 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:07.026 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:06:07.026 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:06:07.026 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:06:07.026 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:07.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.026+0000 7fc769ff4640 1 Processor -- start 2024-03-25T18:06:07.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.026+0000 7fc769ff4640 1 -- start start 2024-03-25T18:06:07.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.026+0000 7fc769ff4640 10 monclient: init 2024-03-25T18:06:07.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.027+0000 7fc769ff4640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:07.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.027+0000 7fc769ff4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:06:07.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.027+0000 7fc769ff4640 10 monclient(hunting): picked mon.noname-c con 0x7fc7640a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:06:07.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.027+0000 7fc769ff4640 10 monclient(hunting): picked mon.noname-a con 0x7fc76413f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:06:07.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.027+0000 7fc769ff4640 10 monclient(hunting): picked mon.noname-b con 0x7fc7641430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:06:07.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.027+0000 7fc769ff4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc764142520 con 0x7fc76413f710 2024-03-25T18:06:07.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.027+0000 7fc769ff4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7640b28a0 con 0x7fc7640a98f0 2024-03-25T18:06:07.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.027+0000 7fc769ff4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7640b2a30 con 0x7fc7641430c0 2024-03-25T18:06:07.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.027+0000 7fc769ff4640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:07.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.027+0000 7fc769ff4640 10 monclient(hunting): authenticate will time out at 7495.604964s 2024-03-25T18:06:07.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.030+0000 7fc763fff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fc7641430c0 0x7fc76414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:50126/0 (socket says 172.21.15.18:50126) 2024-03-25T18:06:07.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.030+0000 7fc763fff640 1 -- 172.21.15.18:0/1149676945 learned_addr learned my addr 172.21.15.18:0/1149676945 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:06:07.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.050+0000 7fc7627fc640 1 -- 172.21.15.18:0/1149676945 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227920951 0 0) 0x7fc7640b2a30 con 0x7fc7641430c0 2024-03-25T18:06:07.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.050+0000 7fc7627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:07.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.050+0000 7fc7627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:07.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.050+0000 7fc7627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:07.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.050+0000 7fc7627fc640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:06:07.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.050+0000 7fc7627fc640 1 -- 172.21.15.18:0/1149676945 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc750003520 con 0x7fc7641430c0 2024-03-25T18:06:07.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.053+0000 7fc7627fc640 1 -- 172.21.15.18:0/1149676945 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625796962 0 0) 0x7fc7640b28a0 con 0x7fc7640a98f0 2024-03-25T18:06:07.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.053+0000 7fc7627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:07.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.053+0000 7fc7627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:07.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.053+0000 7fc7627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:07.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.053+0000 7fc7627fc640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:06:07.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.053+0000 7fc7627fc640 1 -- 172.21.15.18:0/1149676945 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7640b2a30 con 0x7fc7640a98f0 2024-03-25T18:06:07.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.061+0000 7fc7627fc640 1 -- 172.21.15.18:0/1149676945 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302550808 0 0) 0x7fc764142520 con 0x7fc76413f710 2024-03-25T18:06:07.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.061+0000 7fc7627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:07.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.061+0000 7fc7627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:07.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.061+0000 7fc7627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:07.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.061+0000 7fc7627fc640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:06:07.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.061+0000 7fc7627fc640 1 -- 172.21.15.18:0/1149676945 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7640b28a0 con 0x7fc76413f710 2024-03-25T18:06:07.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc7627fc640 1 -- 172.21.15.18:0/1149676945 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 160745103 0 0) 0x7fc7640b28a0 con 0x7fc76413f710 2024-03-25T18:06:07.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc7627fc640 1 -- 172.21.15.18:0/1149676945 >> v1:172.21.15.18:6790/0 conn(0x7fc7641430c0 legacy=0x7fc76414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:07.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc7627fc640 1 -- 172.21.15.18:0/1149676945 >> v1:172.21.15.169:6789/0 conn(0x7fc7640a98f0 legacy=0x7fc7640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:07.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc7627fc640 10 monclient: _finish_hunting 0 2024-03-25T18:06:07.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc7627fc640 1 monclient: found mon.noname-a 2024-03-25T18:06:07.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc7627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:07.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc7627fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:06:07.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc7627fc640 1 -- 172.21.15.18:0/1149676945 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7640b44c0 con 0x7fc76413f710 2024-03-25T18:06:07.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc7627fc640 10 monclient: _finish_auth 0 2024-03-25T18:06:07.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc7627fc640 10 monclient: _check_auth_tickets 2024-03-25T18:06:07.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc7627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:07.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc769ff4640 5 monclient: authenticate success, global_id 4662 2024-03-25T18:06:07.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc7627fc640 1 -- 172.21.15.18:0/1149676945 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc74c0015b0 con 0x7fc76413f710 2024-03-25T18:06:07.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc769ff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:07.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc7627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:07.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc7627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:06:07.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc7627fc640 10 monclient: dump: 2024-03-25T18:06:07.083 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:07.083 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:07.083 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:07.083 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:07.083 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:07.083 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:07.083 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:07.083 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:07.083 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:07.083 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:07.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.081+0000 7fc769ff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:07.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.152+0000 7fc7627fc640 1 -- 172.21.15.18:0/1149676945 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc74c001710 con 0x7fc76413f710 2024-03-25T18:06:07.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.152+0000 7fc7627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:07.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.152+0000 7fc7627fc640 1 -- 172.21.15.18:0/1149676945 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc74c001b60 con 0x7fc76413f710 2024-03-25T18:06:07.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.152+0000 7fc7627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:07.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.152+0000 7fc7627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:06:07.152 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.152+0000 7fc7627fc640 10 monclient: dump: 2024-03-25T18:06:07.153 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:07.153 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:07.153 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:07.153 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:07.153 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:07.153 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:07.153 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:07.153 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:07.153 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:07.153 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:07.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.152+0000 7fc769ff4640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:07.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.152+0000 7fc769ff4640 4 set_mon_vals no callback set 2024-03-25T18:06:07.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.152+0000 7fc769ff4640 10 monclient: shutdown 2024-03-25T18:06:07.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.152+0000 7fc769ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:07.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.152+0000 7fc769ff4640 1 -- 172.21.15.18:0/1149676945 >> v1:172.21.15.18:6789/0 conn(0x7fc76413f710 legacy=0x7fc7641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:07.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.153+0000 7fc769ff4640 1 -- 172.21.15.18:0/1149676945 shutdown_connections 2024-03-25T18:06:07.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.153+0000 7fc769ff4640 1 -- 172.21.15.18:0/1149676945 >> 172.21.15.18:0/1149676945 conn(0x7fc7640b7ea0 msgr2=0x7fc7640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:07.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.153+0000 7fc769ff4640 1 -- 172.21.15.18:0/1149676945 shutdown_connections 2024-03-25T18:06:07.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.153+0000 7fc769ff4640 1 -- 172.21.15.18:0/1149676945 wait complete. 2024-03-25T18:06:07.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.154+0000 7fc769ff4640 10 monclient: build_initial_monmap 2024-03-25T18:06:07.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.154+0000 7fc769ff4640 10 monclient: monmap: 2024-03-25T18:06:07.156 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:06:07.156 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:07.156 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:06:07.156 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:06:07.156 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:07.156 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:06:07.156 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:06:07.156 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:06:07.156 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:06:07.156 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:07.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.154+0000 7fc769ff4640 1 librados: starting msgr at 2024-03-25T18:06:07.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.154+0000 7fc769ff4640 1 librados: starting objecter 2024-03-25T18:06:07.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.154+0000 7fc769ff4640 1 Processor -- start 2024-03-25T18:06:07.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.154+0000 7fc769ff4640 1 -- start start 2024-03-25T18:06:07.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.154+0000 7fc769ff4640 1 librados: setting wanted keys 2024-03-25T18:06:07.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.154+0000 7fc769ff4640 1 librados: calling monclient init 2024-03-25T18:06:07.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.154+0000 7fc769ff4640 10 monclient: init 2024-03-25T18:06:07.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.155+0000 7fc769ff4640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:07.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.155+0000 7fc769ff4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:06:07.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.155+0000 7fc769ff4640 10 monclient(hunting): picked mon.noname-b con 0x7fc7640a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:06:07.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.155+0000 7fc769ff4640 10 monclient(hunting): picked mon.noname-a con 0x7fc76413f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:06:07.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.155+0000 7fc769ff4640 10 monclient(hunting): picked mon.noname-c con 0x7fc7641430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:06:07.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.155+0000 7fc769ff4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7640abd30 con 0x7fc76413f710 2024-03-25T18:06:07.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.155+0000 7fc769ff4640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7641d51f0 con 0x7fc7640a98f0 2024-03-25T18:06:07.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.155+0000 7fc769ff4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7641d77a0 con 0x7fc7641430c0 2024-03-25T18:06:07.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.155+0000 7fc769ff4640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:07.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.155+0000 7fc769ff4640 10 monclient(hunting): authenticate will time out at 7495.733559s 2024-03-25T18:06:07.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.186+0000 7fc762ffd640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fc76413f710 0x7fc7641d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:59938/0 (socket says 172.21.15.18:59938) 2024-03-25T18:06:07.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.186+0000 7fc762ffd640 1 -- 172.21.15.18:0/3487636514 learned_addr learned my addr 172.21.15.18:0/3487636514 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:06:07.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.187+0000 7fc75bfff640 10 client.?.objecter ms_handle_connect 0x7fc76413f710 2024-03-25T18:06:07.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.187+0000 7fc75bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:07.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.196+0000 7fc75bfff640 10 client.?.objecter ms_handle_connect 0x7fc7640a98f0 2024-03-25T18:06:07.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.196+0000 7fc75bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:07.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.205+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275045379 0 0) 0x7fc7641d51f0 con 0x7fc7640a98f0 2024-03-25T18:06:07.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.205+0000 7fc75bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:07.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.205+0000 7fc75bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:07.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.205+0000 7fc75bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:07.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.205+0000 7fc75bfff640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:06:07.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.205+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc744004eb0 con 0x7fc7640a98f0 2024-03-25T18:06:07.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.205+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946225260 0 0) 0x7fc7640abd30 con 0x7fc76413f710 2024-03-25T18:06:07.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.205+0000 7fc75bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:07.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.205+0000 7fc75bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:07.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.205+0000 7fc75bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:07.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.205+0000 7fc75bfff640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:06:07.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.205+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7641d51f0 con 0x7fc76413f710 2024-03-25T18:06:07.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.218+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 545156683 0 0) 0x7fc744004eb0 con 0x7fc7640a98f0 2024-03-25T18:06:07.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.218+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7640abd30 con 0x7fc7640a98f0 2024-03-25T18:06:07.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.218+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc754001930 con 0x7fc7640a98f0 2024-03-25T18:06:07.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.218+0000 7fc75bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:07.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.218+0000 7fc75bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:07.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.218+0000 7fc75bfff640 10 monclient(hunting): dump: 2024-03-25T18:06:07.219 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:07.219 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:07.219 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:07.219 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:07.219 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:07.219 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:07.219 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:07.219 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:07.219 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:07.219 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:07.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.218+0000 7fc75bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:07.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.221+0000 7fc75bfff640 10 client.?.objecter ms_handle_connect 0x7fc7641430c0 2024-03-25T18:06:07.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.221+0000 7fc75bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:07.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.221+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4282595153 0 0) 0x7fc7641d51f0 con 0x7fc76413f710 2024-03-25T18:06:07.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.221+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc744004eb0 con 0x7fc76413f710 2024-03-25T18:06:07.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.221+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc74c001ae0 con 0x7fc76413f710 2024-03-25T18:06:07.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.221+0000 7fc75bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:07.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.221+0000 7fc75bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:06:07.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.221+0000 7fc75bfff640 10 monclient(hunting): dump: 2024-03-25T18:06:07.222 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:07.222 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:07.222 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:07.222 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:07.222 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:07.223 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:07.223 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:07.223 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:07.223 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:07.223 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:07.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1445049901 0 0) 0x7fc744004eb0 con 0x7fc76413f710 2024-03-25T18:06:07.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 >> v1:172.21.15.18:6790/0 conn(0x7fc7641430c0 legacy=0x7fc7641e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:07.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 >> v1:172.21.15.169:6789/0 conn(0x7fc7640a98f0 legacy=0x7fc764138740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:07.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc75bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:06:07.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc75bfff640 1 monclient: found mon.a 2024-03-25T18:06:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc75bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc75bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:06:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc764138e50 con 0x7fc76413f710 2024-03-25T18:06:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc75bfff640 10 monclient: _finish_auth 0 2024-03-25T18:06:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc75bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:06:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc75bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc769ff4640 5 monclient: authenticate success, global_id 4665 2024-03-25T18:06:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc769ff4640 10 monclient: _renew_subs 2024-03-25T18:06:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc769ff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:06:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc769ff4640 1 -- 172.21.15.18:0/3487636514 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7641d83d0 con 0x7fc76413f710 2024-03-25T18:06:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc769ff4640 10 client.4665.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc769ff4640 10 monclient: _renew_subs 2024-03-25T18:06:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc769ff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:06:07.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc769ff4640 1 -- 172.21.15.18:0/3487636514 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7641381b0 con 0x7fc76413f710 2024-03-25T18:06:07.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.239+0000 7fc769ff4640 1 librados: init done 2024-03-25T18:06:07.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.240+0000 7fc769ff4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:07.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.240+0000 7fc769ff4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:07.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.240+0000 7fc769ff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:06:07.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.240+0000 7fc769ff4640 1 -- 172.21.15.18:0/3487636514 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7641d7600 con 0x7fc76413f710 2024-03-25T18:06:07.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.571+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc74c0038a0 con 0x7fc76413f710 2024-03-25T18:06:07.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.571+0000 7fc75bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:07.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.571+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc74c003ad0 con 0x7fc76413f710 2024-03-25T18:06:07.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.571+0000 7fc75bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:07.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.571+0000 7fc75bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:06:07.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.571+0000 7fc75bfff640 10 monclient: dump: 2024-03-25T18:06:07.572 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:07.572 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:07.572 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:07.572 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:07.572 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:07.572 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:07.572 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:07.573 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:07.573 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:07.573 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:07.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.571+0000 7fc761ffb640 4 set_mon_vals no callback set 2024-03-25T18:06:07.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.572+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fc74c003cf0 con 0x7fc76413f710 2024-03-25T18:06:07.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.572+0000 7fc7637fe640 10 monclient: get_auth_request con 0x7fc74406dee0 auth_method 0 2024-03-25T18:06:07.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.573+0000 7fc75bfff640 10 client.4665.objecter ms_handle_connect 0x7fc74406dee0 2024-03-25T18:06:07.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.580+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7fc74c0546b0 con 0x7fc76413f710 2024-03-25T18:06:07.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.580+0000 7fc75bfff640 10 client.4665.objecter ms_dispatch 0x7fc764000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:07.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.580+0000 7fc75bfff640 3 client.4665.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:07.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.580+0000 7fc75bfff640 3 client.4665.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:07.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.580+0000 7fc75bfff640 20 client.4665.objecter dump_active .. 0 homeless 2024-03-25T18:06:07.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.580+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc74c0501c0 con 0x7fc76413f710 2024-03-25T18:06:07.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.580+0000 7fc75bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:07.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.580+0000 7fc75bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:07.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.837+0000 7fc769ff4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:07.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.837+0000 7fc769ff4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:07.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.837+0000 7fc769ff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:06:07.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.837+0000 7fc769ff4640 1 -- 172.21.15.18:0/3487636514 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7640627b0 con 0x7fc76413f710 2024-03-25T18:06:07.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.875+0000 7fc75bfff640 1 -- 172.21.15.18:0/3487636514 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7fc74c0520f0 con 0x7fc76413f710 2024-03-25T18:06:07.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.875+0000 7fc75bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:07.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.875+0000 7fc75bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:07.875 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:06:07.875 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:07.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.877+0000 7fc769ff4640 10 librados: watch_flush enter 2024-03-25T18:06:07.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.877+0000 7fc769ff4640 10 librados: watch_flush exit 2024-03-25T18:06:07.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.877+0000 7fc769ff4640 20 client.4665.objecter shutdown clearing up homeless session... 2024-03-25T18:06:07.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.877+0000 7fc769ff4640 10 client.4665.objecter successfully canceled tick 2024-03-25T18:06:07.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.878+0000 7fc769ff4640 1 -- 172.21.15.18:0/3487636514 >> v1:172.21.15.18:6800/3721244451 conn(0x7fc74406dee0 legacy=0x7fc7440703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:07.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.878+0000 7fc769ff4640 10 monclient: shutdown 2024-03-25T18:06:07.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.878+0000 7fc769ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:07.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.878+0000 7fc769ff4640 1 -- 172.21.15.18:0/3487636514 >> v1:172.21.15.18:6789/0 conn(0x7fc76413f710 legacy=0x7fc7641d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:07.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.878+0000 7fc769ff4640 1 -- 172.21.15.18:0/3487636514 shutdown_connections 2024-03-25T18:06:07.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.878+0000 7fc769ff4640 1 -- 172.21.15.18:0/3487636514 >> 172.21.15.18:0/3487636514 conn(0x7fc7640b7ea0 msgr2=0x7fc764074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:07.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.878+0000 7fc769ff4640 1 -- 172.21.15.18:0/3487636514 shutdown_connections 2024-03-25T18:06:07.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.878+0000 7fc769ff4640 1 -- 172.21.15.18:0/3487636514 wait complete. 2024-03-25T18:06:07.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:07.878+0000 7fc769ff4640 1 librados: shutdown 2024-03-25T18:06:07.890 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:09.013 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:06.805 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:09.020 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:06.813 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:09.051 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:06.844 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:09.076 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:06.825 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:09.086 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:06.879 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:09.090 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:06.838 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:09.097 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:06.889 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:09.130 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:06.879 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:09.131 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:06.924 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:14.696 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:14.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.815+0000 7f072ebe9640 10 monclient: get_monmap_and_config 2024-03-25T18:06:14.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.815+0000 7f072ebe9640 10 monclient: build_initial_monmap 2024-03-25T18:06:14.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.815+0000 7f072ebe9640 10 monclient: monmap: 2024-03-25T18:06:14.815 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:06:14.815 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:14.815 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:06:14.815510+0000 2024-03-25T18:06:14.815 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:06:14.815510+0000 2024-03-25T18:06:14.815 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:14.816 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:14.816 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:06:14.816 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:06:14.816 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:06:14.816 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:14.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.816+0000 7f072ebe9640 1 Processor -- start 2024-03-25T18:06:14.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.816+0000 7f072ebe9640 1 -- start start 2024-03-25T18:06:14.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.816+0000 7f072ebe9640 10 monclient: init 2024-03-25T18:06:14.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.816+0000 7f072ebe9640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:14.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.816+0000 7f072ebe9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:06:14.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.816+0000 7f072ebe9640 10 monclient(hunting): picked mon.noname-a con 0x7f07280a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:06:14.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.816+0000 7f072ebe9640 10 monclient(hunting): picked mon.noname-c con 0x7f072813f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:06:14.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.816+0000 7f072ebe9640 10 monclient(hunting): picked mon.noname-b con 0x7f07281430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:06:14.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.816+0000 7f072ebe9640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0728142520 con 0x7f07280a98f0 2024-03-25T18:06:14.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.817+0000 7f072ebe9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07280b28a0 con 0x7f072813f710 2024-03-25T18:06:14.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.817+0000 7f072ebe9640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07280b2a30 con 0x7f07281430c0 2024-03-25T18:06:14.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.817+0000 7f072ebe9640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:14.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.817+0000 7f072ebe9640 10 monclient(hunting): authenticate will time out at 7503.394676s 2024-03-25T18:06:14.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.825+0000 7f072c985640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f07280a98f0 0x7f07280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:47368/0 (socket says 172.21.15.18:47368) 2024-03-25T18:06:14.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.825+0000 7f072c985640 1 -- 172.21.15.18:0/1693497073 learned_addr learned my addr 172.21.15.18:0/1693497073 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:06:14.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.834+0000 7f07277fe640 1 -- 172.21.15.18:0/1693497073 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 501411157 0 0) 0x7f07280b2a30 con 0x7f07281430c0 2024-03-25T18:06:14.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.834+0000 7f07277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:14.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.834+0000 7f07277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:14.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.834+0000 7f07277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:14.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.834+0000 7f07277fe640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:06:14.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.834+0000 7f07277fe640 1 -- 172.21.15.18:0/1693497073 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0718003520 con 0x7f07281430c0 2024-03-25T18:06:14.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.842+0000 7f07277fe640 1 -- 172.21.15.18:0/1693497073 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734827810 0 0) 0x7f07280b28a0 con 0x7f072813f710 2024-03-25T18:06:14.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.842+0000 7f07277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:14.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.842+0000 7f07277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:14.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.842+0000 7f07277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:14.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.842+0000 7f07277fe640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:06:14.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.842+0000 7f07277fe640 1 -- 172.21.15.18:0/1693497073 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07280b2a30 con 0x7f072813f710 2024-03-25T18:06:14.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.844+0000 7f07277fe640 1 -- 172.21.15.18:0/1693497073 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755769899 0 0) 0x7f0728142520 con 0x7f07280a98f0 2024-03-25T18:06:14.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.844+0000 7f07277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:14.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.844+0000 7f07277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:14.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.844+0000 7f07277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:14.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.844+0000 7f07277fe640 10 monclient(hunting): my global_id is 4668 2024-03-25T18:06:14.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.844+0000 7f07277fe640 1 -- 172.21.15.18:0/1693497073 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07280b28a0 con 0x7f07280a98f0 2024-03-25T18:06:14.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f07277fe640 1 -- 172.21.15.18:0/1693497073 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 111563745 0 0) 0x7f0718003520 con 0x7f07281430c0 2024-03-25T18:06:14.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f07277fe640 1 -- 172.21.15.18:0/1693497073 >> v1:172.21.15.169:6789/0 conn(0x7f072813f710 legacy=0x7f07281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:14.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f07277fe640 1 -- 172.21.15.18:0/1693497073 >> v1:172.21.15.18:6789/0 conn(0x7f07280a98f0 legacy=0x7f07280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:14.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f07277fe640 10 monclient: _finish_hunting 0 2024-03-25T18:06:14.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f07277fe640 1 monclient: found mon.noname-b 2024-03-25T18:06:14.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f07277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:14.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f07277fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:06:14.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f07277fe640 1 -- 172.21.15.18:0/1693497073 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07280b44c0 con 0x7f07281430c0 2024-03-25T18:06:14.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f07277fe640 10 monclient: _finish_auth 0 2024-03-25T18:06:14.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f07277fe640 10 monclient: _check_auth_tickets 2024-03-25T18:06:14.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f07277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:14.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f07277fe640 1 -- 172.21.15.18:0/1693497073 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0720001a10 con 0x7f07281430c0 2024-03-25T18:06:14.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f072ebe9640 5 monclient: authenticate success, global_id 4676 2024-03-25T18:06:14.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f072ebe9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:14.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f07277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:14.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f07277fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:14.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f07277fe640 10 monclient: dump: 2024-03-25T18:06:14.851 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:14.851 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:14.851 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:14.851 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:14.851 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:14.851 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:14.851 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:14.851 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:14.851 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:14.851 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:14.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.849+0000 7f072ebe9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:14.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.871+0000 7f07277fe640 1 -- 172.21.15.18:0/1693497073 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0720001b70 con 0x7f07281430c0 2024-03-25T18:06:14.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.871+0000 7f07277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:14.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.871+0000 7f07277fe640 1 -- 172.21.15.18:0/1693497073 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f0720001f80 con 0x7f07281430c0 2024-03-25T18:06:14.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.871+0000 7f07277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:14.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.871+0000 7f07277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:06:14.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.871+0000 7f07277fe640 10 monclient: dump: 2024-03-25T18:06:14.872 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:14.872 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:14.872 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:14.872 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:14.872 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:14.872 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:14.872 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:14.872 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:14.872 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:14.872 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:14.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.871+0000 7f072ebe9640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:14.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.871+0000 7f072ebe9640 4 set_mon_vals no callback set 2024-03-25T18:06:14.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.872+0000 7f072ebe9640 10 monclient: shutdown 2024-03-25T18:06:14.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.872+0000 7f072ebe9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:14.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.872+0000 7f072ebe9640 1 -- 172.21.15.18:0/1693497073 >> v1:172.21.15.18:6790/0 conn(0x7f07281430c0 legacy=0x7f072814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:14.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.872+0000 7f072ebe9640 1 -- 172.21.15.18:0/1693497073 shutdown_connections 2024-03-25T18:06:14.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.872+0000 7f072ebe9640 1 -- 172.21.15.18:0/1693497073 >> 172.21.15.18:0/1693497073 conn(0x7f07280b7ea0 msgr2=0x7f07280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:14.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.872+0000 7f072ebe9640 1 -- 172.21.15.18:0/1693497073 shutdown_connections 2024-03-25T18:06:14.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.872+0000 7f072ebe9640 1 -- 172.21.15.18:0/1693497073 wait complete. 2024-03-25T18:06:14.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.873+0000 7f072ebe9640 10 monclient: build_initial_monmap 2024-03-25T18:06:14.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.873+0000 7f072ebe9640 10 monclient: monmap: 2024-03-25T18:06:14.875 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:06:14.875 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:14.875 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:06:14.875 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:06:14.875 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:14.875 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:06:14.875 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:06:14.875 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:06:14.875 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:06:14.876 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:14.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.873+0000 7f072ebe9640 1 librados: starting msgr at 2024-03-25T18:06:14.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.873+0000 7f072ebe9640 1 librados: starting objecter 2024-03-25T18:06:14.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.874+0000 7f072ebe9640 1 Processor -- start 2024-03-25T18:06:14.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.874+0000 7f072ebe9640 1 -- start start 2024-03-25T18:06:14.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.874+0000 7f072ebe9640 1 librados: setting wanted keys 2024-03-25T18:06:14.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.874+0000 7f072ebe9640 1 librados: calling monclient init 2024-03-25T18:06:14.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.874+0000 7f072ebe9640 10 monclient: init 2024-03-25T18:06:14.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.874+0000 7f072ebe9640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:14.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.874+0000 7f072ebe9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:06:14.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.874+0000 7f072ebe9640 10 monclient(hunting): picked mon.noname-b con 0x7f07280a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:06:14.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.874+0000 7f072ebe9640 10 monclient(hunting): picked mon.noname-a con 0x7f072813f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:06:14.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.875+0000 7f072ebe9640 10 monclient(hunting): picked mon.noname-c con 0x7f07281430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:06:14.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.875+0000 7f072ebe9640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07281daca0 con 0x7f072813f710 2024-03-25T18:06:14.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.875+0000 7f072ebe9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0728138ab0 con 0x7f07280a98f0 2024-03-25T18:06:14.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.875+0000 7f072ebe9640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07280aba90 con 0x7f07281430c0 2024-03-25T18:06:14.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.875+0000 7f072ebe9640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:14.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.875+0000 7f072ebe9640 10 monclient(hunting): authenticate will time out at 7503.452708s 2024-03-25T18:06:14.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.890+0000 7f072c985640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f07280a98f0 0x7f07281383a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:60410/0 (socket says 172.21.15.18:60410) 2024-03-25T18:06:14.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.890+0000 7f072c985640 1 -- 172.21.15.18:0/4132740982 learned_addr learned my addr 172.21.15.18:0/4132740982 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:06:14.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.892+0000 7f07257fa640 10 client.?.objecter ms_handle_connect 0x7f07280a98f0 2024-03-25T18:06:14.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.892+0000 7f07257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:14.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.896+0000 7f07257fa640 10 client.?.objecter ms_handle_connect 0x7f07281430c0 2024-03-25T18:06:14.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.896+0000 7f07257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:14.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.898+0000 7f07257fa640 10 client.?.objecter ms_handle_connect 0x7f072813f710 2024-03-25T18:06:14.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.898+0000 7f07257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:14.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.898+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3944927088 0 0) 0x7f0728138ab0 con 0x7f07280a98f0 2024-03-25T18:06:14.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.898+0000 7f07257fa640 10 monclient(hunting): _negotiate 2024-03-25T18:06:14.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.898+0000 7f07257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:14.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.898+0000 7f07257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:14.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.898+0000 7f07257fa640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:06:14.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.898+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f070c004eb0 con 0x7f07280a98f0 2024-03-25T18:06:14.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.906+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599058376 0 0) 0x7f07280aba90 con 0x7f07281430c0 2024-03-25T18:06:14.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.906+0000 7f07257fa640 10 monclient(hunting): _negotiate 2024-03-25T18:06:14.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.906+0000 7f07257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:14.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.906+0000 7f07257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:14.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.906+0000 7f07257fa640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:06:14.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.906+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0728138ab0 con 0x7f07281430c0 2024-03-25T18:06:14.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.910+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1825042944 0 0) 0x7f070c004eb0 con 0x7f07280a98f0 2024-03-25T18:06:14.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.910+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07280aba90 con 0x7f07280a98f0 2024-03-25T18:06:14.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.910+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f071c001910 con 0x7f07280a98f0 2024-03-25T18:06:14.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.910+0000 7f07257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:14.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.910+0000 7f07257fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:14.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.910+0000 7f07257fa640 10 monclient(hunting): dump: 2024-03-25T18:06:14.911 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:14.911 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:14.911 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:14.911 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:14.911 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:14.911 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:14.911 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:14.911 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:14.911 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:14.911 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:14.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.910+0000 7f07257fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.913+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404979261 0 0) 0x7f07281daca0 con 0x7f072813f710 2024-03-25T18:06:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.913+0000 7f07257fa640 10 monclient(hunting): _negotiate 2024-03-25T18:06:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.913+0000 7f07257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.913+0000 7f07257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.913+0000 7f07257fa640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:06:14.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.913+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f070c004eb0 con 0x7f072813f710 2024-03-25T18:06:14.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.921+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 332127456 0 0) 0x7f07280aba90 con 0x7f07280a98f0 2024-03-25T18:06:14.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.921+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 >> v1:172.21.15.18:6790/0 conn(0x7f07281430c0 legacy=0x7f07281d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:14.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.921+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 >> v1:172.21.15.18:6789/0 conn(0x7f072813f710 legacy=0x7f07281db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:14.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.922+0000 7f07257fa640 10 monclient: _finish_hunting 0 2024-03-25T18:06:14.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.922+0000 7f07257fa640 1 monclient: found mon.b 2024-03-25T18:06:14.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.922+0000 7f07257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:14.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.922+0000 7f07257fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:14.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.922+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07281d41e0 con 0x7f07280a98f0 2024-03-25T18:06:14.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.922+0000 7f07257fa640 10 monclient: _finish_auth 0 2024-03-25T18:06:14.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.922+0000 7f07257fa640 10 monclient: _check_auth_tickets 2024-03-25T18:06:14.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.922+0000 7f07257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:14.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.922+0000 7f072ebe9640 5 monclient: authenticate success, global_id 4660 2024-03-25T18:06:14.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.922+0000 7f072ebe9640 10 monclient: _renew_subs 2024-03-25T18:06:14.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.922+0000 7f072ebe9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:14.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.922+0000 7f072ebe9640 1 -- 172.21.15.18:0/4132740982 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07281db840 con 0x7f07280a98f0 2024-03-25T18:06:14.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.922+0000 7f072ebe9640 10 client.4660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:14.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.922+0000 7f072ebe9640 10 monclient: _renew_subs 2024-03-25T18:06:14.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.922+0000 7f072ebe9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:14.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.922+0000 7f072ebe9640 1 -- 172.21.15.18:0/4132740982 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0728138f00 con 0x7f07280a98f0 2024-03-25T18:06:14.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.922+0000 7f072ebe9640 1 librados: init done 2024-03-25T18:06:14.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.923+0000 7f072ebe9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:14.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.923+0000 7f072ebe9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:14.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.923+0000 7f072ebe9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:14.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:14.923+0000 7f072ebe9640 1 -- 172.21.15.18:0/4132740982 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07280627b0 con 0x7f07280a98f0 2024-03-25T18:06:15.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.205+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f071c0047e0 con 0x7f07280a98f0 2024-03-25T18:06:15.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.205+0000 7f07257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:15.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.205+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f071c004bf0 con 0x7f07280a98f0 2024-03-25T18:06:15.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.205+0000 7f07257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:15.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.205+0000 7f07257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:15.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.205+0000 7f07257fa640 10 monclient: dump: 2024-03-25T18:06:15.206 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:15.206 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:15.206 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:15.206 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:15.206 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:15.206 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:15.206 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:15.206 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:15.207 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:15.207 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:15.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.205+0000 7f0726ffd640 4 set_mon_vals no callback set 2024-03-25T18:06:15.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.206+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f071c004dd0 con 0x7f07280a98f0 2024-03-25T18:06:15.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.207+0000 7f0727fff640 10 monclient: get_auth_request con 0x7f070c06dd60 auth_method 0 2024-03-25T18:06:15.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.207+0000 7f07257fa640 10 client.4660.objecter ms_handle_connect 0x7f070c06dd60 2024-03-25T18:06:15.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.211+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f071c0548d0 con 0x7f07280a98f0 2024-03-25T18:06:15.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.211+0000 7f07257fa640 10 client.4660.objecter ms_dispatch 0x7f0728000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:15.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.211+0000 7f07257fa640 3 client.4660.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:15.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.211+0000 7f07257fa640 3 client.4660.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:15.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.211+0000 7f07257fa640 20 client.4660.objecter dump_active .. 0 homeless 2024-03-25T18:06:15.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.215+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f071c0503e0 con 0x7f07280a98f0 2024-03-25T18:06:15.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.215+0000 7f07257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:15.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.215+0000 7f07257fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:15.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.448+0000 7f072ebe9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:15.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.448+0000 7f072ebe9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:15.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.448+0000 7f072ebe9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:15.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.448+0000 7f072ebe9640 1 -- 172.21.15.18:0/4132740982 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07281d4370 con 0x7f07280a98f0 2024-03-25T18:06:15.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.590+0000 7f07257fa640 1 -- 172.21.15.18:0/4132740982 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f071c052310 con 0x7f07280a98f0 2024-03-25T18:06:15.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.590+0000 7f07257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:15.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.590+0000 7f07257fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:15.591 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:06:15.591 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:15.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.592+0000 7f072ebe9640 10 librados: watch_flush enter 2024-03-25T18:06:15.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.593+0000 7f072ebe9640 10 librados: watch_flush exit 2024-03-25T18:06:15.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.593+0000 7f072ebe9640 20 client.4660.objecter shutdown clearing up homeless session... 2024-03-25T18:06:15.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.593+0000 7f072ebe9640 10 client.4660.objecter successfully canceled tick 2024-03-25T18:06:15.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.593+0000 7f072ebe9640 1 -- 172.21.15.18:0/4132740982 >> v1:172.21.15.18:6800/3721244451 conn(0x7f070c06dd60 legacy=0x7f070c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:15.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.593+0000 7f072ebe9640 10 monclient: shutdown 2024-03-25T18:06:15.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.593+0000 7f072ebe9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:15.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.593+0000 7f072ebe9640 1 -- 172.21.15.18:0/4132740982 >> v1:172.21.15.169:6789/0 conn(0x7f07280a98f0 legacy=0x7f07281383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:15.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.593+0000 7f072ebe9640 1 -- 172.21.15.18:0/4132740982 shutdown_connections 2024-03-25T18:06:15.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.593+0000 7f072ebe9640 1 -- 172.21.15.18:0/4132740982 >> 172.21.15.18:0/4132740982 conn(0x7f07280b7ea0 msgr2=0x7f0728074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:15.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.593+0000 7f072ebe9640 1 -- 172.21.15.18:0/4132740982 shutdown_connections 2024-03-25T18:06:15.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.593+0000 7f072ebe9640 1 -- 172.21.15.18:0/4132740982 wait complete. 2024-03-25T18:06:15.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:15.593+0000 7f072ebe9640 1 librados: shutdown 2024-03-25T18:06:15.604 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:22.410 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:22.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.524+0000 7f66b1ec1640 10 monclient: get_monmap_and_config 2024-03-25T18:06:22.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.524+0000 7f66b1ec1640 10 monclient: build_initial_monmap 2024-03-25T18:06:22.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.524+0000 7f66b1ec1640 10 monclient: monmap: 2024-03-25T18:06:22.525 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:06:22.525 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:22.525 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:06:22.524895+0000 2024-03-25T18:06:22.525 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:06:22.524895+0000 2024-03-25T18:06:22.525 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:22.525 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:22.525 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:06:22.525 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:06:22.525 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:06:22.525 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:22.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.525+0000 7f66b1ec1640 1 Processor -- start 2024-03-25T18:06:22.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.525+0000 7f66b1ec1640 1 -- start start 2024-03-25T18:06:22.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.525+0000 7f66b1ec1640 10 monclient: init 2024-03-25T18:06:22.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.526+0000 7f66b1ec1640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:22.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.526+0000 7f66b1ec1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:06:22.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.526+0000 7f66b1ec1640 10 monclient(hunting): picked mon.noname-c con 0x7f66ac0a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:06:22.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.526+0000 7f66b1ec1640 10 monclient(hunting): picked mon.noname-b con 0x7f66ac13f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:06:22.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.526+0000 7f66b1ec1640 10 monclient(hunting): picked mon.noname-a con 0x7f66ac1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:06:22.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.526+0000 7f66b1ec1640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ac142520 con 0x7f66ac1430c0 2024-03-25T18:06:22.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.526+0000 7f66b1ec1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ac0b28a0 con 0x7f66ac0a98f0 2024-03-25T18:06:22.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.526+0000 7f66b1ec1640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ac0b2a30 con 0x7f66ac13f710 2024-03-25T18:06:22.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.526+0000 7f66b1ec1640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:22.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.526+0000 7f66b1ec1640 10 monclient(hunting): authenticate will time out at 7511.104036s 2024-03-25T18:06:22.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.529+0000 7f66aaffd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f66ac13f710 0x7f66ac1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:57928/0 (socket says 172.21.15.18:57928) 2024-03-25T18:06:22.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.529+0000 7f66aaffd640 1 -- 172.21.15.18:0/2437565278 learned_addr learned my addr 172.21.15.18:0/2437565278 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:06:22.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.539+0000 7f66aa7fc640 1 -- 172.21.15.18:0/2437565278 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423897496 0 0) 0x7f66ac0b2a30 con 0x7f66ac13f710 2024-03-25T18:06:22.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.539+0000 7f66aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:22.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.539+0000 7f66aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:22.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.539+0000 7f66aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:22.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.539+0000 7f66aa7fc640 10 monclient(hunting): my global_id is 4682 2024-03-25T18:06:22.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.539+0000 7f66aa7fc640 1 -- 172.21.15.18:0/2437565278 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6694003520 con 0x7f66ac13f710 2024-03-25T18:06:22.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.550+0000 7f66aa7fc640 1 -- 172.21.15.18:0/2437565278 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3968349160 0 0) 0x7f6694003520 con 0x7f66ac13f710 2024-03-25T18:06:22.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.550+0000 7f66aa7fc640 1 -- 172.21.15.18:0/2437565278 >> v1:172.21.15.169:6789/0 conn(0x7f66ac0a98f0 legacy=0x7f66ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:22.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.550+0000 7f66aa7fc640 1 -- 172.21.15.18:0/2437565278 >> v1:172.21.15.18:6789/0 conn(0x7f66ac1430c0 legacy=0x7f66ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:22.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.550+0000 7f66aa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:06:22.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.550+0000 7f66aa7fc640 1 monclient: found mon.noname-b 2024-03-25T18:06:22.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.550+0000 7f66aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:22.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.550+0000 7f66aa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:06:22.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.550+0000 7f66aa7fc640 1 -- 172.21.15.18:0/2437565278 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ac0b44c0 con 0x7f66ac13f710 2024-03-25T18:06:22.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.550+0000 7f66aa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:06:22.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.550+0000 7f66aa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:06:22.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.550+0000 7f66aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:22.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.550+0000 7f66aa7fc640 1 -- 172.21.15.18:0/2437565278 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f66a0001690 con 0x7f66ac13f710 2024-03-25T18:06:22.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.550+0000 7f66aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:22.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.550+0000 7f66aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:22.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.550+0000 7f66aa7fc640 10 monclient: dump: 2024-03-25T18:06:22.551 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:22.551 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:22.551 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:22.551 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:22.551 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:22.552 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:22.552 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:22.552 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:22.552 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:22.552 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:22.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.550+0000 7f66b1ec1640 5 monclient: authenticate success, global_id 4682 2024-03-25T18:06:22.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.550+0000 7f66b1ec1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:22.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.560+0000 7f66aa7fc640 1 -- 172.21.15.18:0/2437565278 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66a00017f0 con 0x7f66ac13f710 2024-03-25T18:06:22.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.560+0000 7f66aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:22.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.560+0000 7f66aa7fc640 1 -- 172.21.15.18:0/2437565278 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f66a0001c40 con 0x7f66ac13f710 2024-03-25T18:06:22.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.560+0000 7f66b1ec1640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:22.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.560+0000 7f66b1ec1640 4 set_mon_vals no callback set 2024-03-25T18:06:22.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.560+0000 7f66aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:22.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.560+0000 7f66aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:06:22.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.560+0000 7f66aa7fc640 10 monclient: dump: 2024-03-25T18:06:22.561 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:22.561 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:22.561 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:22.561 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:22.561 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:22.561 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:22.561 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:22.561 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:22.561 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:22.561 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:22.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.561+0000 7f66b1ec1640 10 monclient: shutdown 2024-03-25T18:06:22.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.561+0000 7f66b1ec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:22.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.561+0000 7f66b1ec1640 1 -- 172.21.15.18:0/2437565278 >> v1:172.21.15.18:6790/0 conn(0x7f66ac13f710 legacy=0x7f66ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:22.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.561+0000 7f66b1ec1640 1 -- 172.21.15.18:0/2437565278 shutdown_connections 2024-03-25T18:06:22.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.561+0000 7f66b1ec1640 1 -- 172.21.15.18:0/2437565278 >> 172.21.15.18:0/2437565278 conn(0x7f66ac0b7ea0 msgr2=0x7f66ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:22.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.561+0000 7f66b1ec1640 1 -- 172.21.15.18:0/2437565278 shutdown_connections 2024-03-25T18:06:22.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.561+0000 7f66b1ec1640 1 -- 172.21.15.18:0/2437565278 wait complete. 2024-03-25T18:06:22.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.562+0000 7f66b1ec1640 10 monclient: build_initial_monmap 2024-03-25T18:06:22.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.562+0000 7f66b1ec1640 10 monclient: monmap: 2024-03-25T18:06:22.563 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:06:22.563 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:22.563 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:06:22.563 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:06:22.564 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:22.564 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:06:22.564 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:06:22.564 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:06:22.564 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:06:22.564 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:22.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.562+0000 7f66b1ec1640 1 librados: starting msgr at 2024-03-25T18:06:22.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.562+0000 7f66b1ec1640 1 librados: starting objecter 2024-03-25T18:06:22.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.562+0000 7f66b1ec1640 1 Processor -- start 2024-03-25T18:06:22.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.562+0000 7f66b1ec1640 1 -- start start 2024-03-25T18:06:22.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.562+0000 7f66b1ec1640 1 librados: setting wanted keys 2024-03-25T18:06:22.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.562+0000 7f66b1ec1640 1 librados: calling monclient init 2024-03-25T18:06:22.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.562+0000 7f66b1ec1640 10 monclient: init 2024-03-25T18:06:22.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.563+0000 7f66b1ec1640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:22.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.563+0000 7f66b1ec1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:06:22.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.563+0000 7f66b1ec1640 10 monclient(hunting): picked mon.noname-a con 0x7f66ac0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:06:22.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.563+0000 7f66b1ec1640 10 monclient(hunting): picked mon.noname-b con 0x7f66ac13f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:06:22.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.563+0000 7f66b1ec1640 10 monclient(hunting): picked mon.noname-c con 0x7f66ac1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:06:22.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.563+0000 7f66b1ec1640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ac139a40 con 0x7f66ac0a98f0 2024-03-25T18:06:22.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.563+0000 7f66b1ec1640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ac133f20 con 0x7f66ac13f710 2024-03-25T18:06:22.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.563+0000 7f66b1ec1640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ac0ab950 con 0x7f66ac1430c0 2024-03-25T18:06:22.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.563+0000 7f66b1ec1640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:22.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.563+0000 7f66b1ec1640 10 monclient(hunting): authenticate will time out at 7511.141499s 2024-03-25T18:06:22.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.573+0000 7f66aaffd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f66ac13f710 0x7f66ac133100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:36848/0 (socket says 172.21.15.18:36848) 2024-03-25T18:06:22.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.573+0000 7f66aaffd640 1 -- 172.21.15.18:0/4220618169 learned_addr learned my addr 172.21.15.18:0/4220618169 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:06:22.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.574+0000 7f6693fff640 10 client.?.objecter ms_handle_connect 0x7f66ac13f710 2024-03-25T18:06:22.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.574+0000 7f6693fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:22.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.581+0000 7f6693fff640 10 client.?.objecter ms_handle_connect 0x7f66ac0a98f0 2024-03-25T18:06:22.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.581+0000 7f6693fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:22.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.586+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905013699 0 0) 0x7f66ac133f20 con 0x7f66ac13f710 2024-03-25T18:06:22.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.586+0000 7f6693fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:22.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.586+0000 7f6693fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:22.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.586+0000 7f6693fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:22.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.586+0000 7f6693fff640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:06:22.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.586+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6698004eb0 con 0x7f66ac13f710 2024-03-25T18:06:22.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.587+0000 7f6693fff640 10 client.?.objecter ms_handle_connect 0x7f66ac1430c0 2024-03-25T18:06:22.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.587+0000 7f6693fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:22.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.590+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100608100 0 0) 0x7f66ac139a40 con 0x7f66ac0a98f0 2024-03-25T18:06:22.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.590+0000 7f6693fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:22.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.590+0000 7f6693fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:22.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.590+0000 7f6693fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:22.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.590+0000 7f6693fff640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:06:22.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.590+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66ac133f20 con 0x7f66ac0a98f0 2024-03-25T18:06:22.601 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.601+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768707730 0 0) 0x7f66ac0ab950 con 0x7f66ac1430c0 2024-03-25T18:06:22.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.601+0000 7f6693fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:22.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.601+0000 7f6693fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:22.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.601+0000 7f6693fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:22.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.601+0000 7f6693fff640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:06:22.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.601+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66ac139a40 con 0x7f66ac1430c0 2024-03-25T18:06:22.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.604+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4005458862 0 0) 0x7f66ac133f20 con 0x7f66ac0a98f0 2024-03-25T18:06:22.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.604+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66ac0ab950 con 0x7f66ac0a98f0 2024-03-25T18:06:22.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.604+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f669c0018b0 con 0x7f66ac0a98f0 2024-03-25T18:06:22.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.604+0000 7f6693fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:22.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.604+0000 7f6693fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:06:22.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.604+0000 7f6693fff640 10 monclient(hunting): dump: 2024-03-25T18:06:22.605 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:22.605 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:22.606 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:22.606 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:22.606 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:22.606 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:22.606 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:22.606 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:22.606 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:22.606 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:22.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.604+0000 7f6693fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:22.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.610+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2829035062 0 0) 0x7f6698004eb0 con 0x7f66ac13f710 2024-03-25T18:06:22.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.610+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66ac133f20 con 0x7f66ac13f710 2024-03-25T18:06:22.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.610+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f66a0001b00 con 0x7f66ac13f710 2024-03-25T18:06:22.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.610+0000 7f6693fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:22.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.610+0000 7f6693fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:22.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.610+0000 7f6693fff640 10 monclient(hunting): dump: 2024-03-25T18:06:22.611 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:22.611 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:22.611 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:22.611 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:22.611 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:22.611 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:22.611 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:22.611 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:22.611 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:22.611 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:22.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.622+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3431679513 0 0) 0x7f66ac139a40 con 0x7f66ac1430c0 2024-03-25T18:06:22.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.622+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6698004eb0 con 0x7f66ac1430c0 2024-03-25T18:06:22.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.622+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f66a4001a90 con 0x7f66ac1430c0 2024-03-25T18:06:22.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.622+0000 7f6693fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:22.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.622+0000 7f6693fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:06:22.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.622+0000 7f6693fff640 10 monclient(hunting): dump: 2024-03-25T18:06:22.623 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:22.623 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:22.623 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:22.623 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:22.623 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:22.623 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:22.624 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:22.624 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:22.624 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:22.624 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:22.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3104233626 0 0) 0x7f66ac133f20 con 0x7f66ac13f710 2024-03-25T18:06:22.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 >> v1:172.21.15.18:6790/0 conn(0x7f66ac1430c0 legacy=0x7f66ac133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:22.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 >> v1:172.21.15.18:6789/0 conn(0x7f66ac0a98f0 legacy=0x7f66ac138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:22.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f6693fff640 10 monclient: _finish_hunting 0 2024-03-25T18:06:22.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f6693fff640 1 monclient: found mon.b 2024-03-25T18:06:22.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f6693fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:22.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f6693fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:22.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ac0abc30 con 0x7f66ac13f710 2024-03-25T18:06:22.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f6693fff640 10 monclient: _finish_auth 0 2024-03-25T18:06:22.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f6693fff640 10 monclient: _check_auth_tickets 2024-03-25T18:06:22.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f6693fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:22.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f66b1ec1640 5 monclient: authenticate success, global_id 4666 2024-03-25T18:06:22.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f66b1ec1640 10 monclient: _renew_subs 2024-03-25T18:06:22.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f66b1ec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:22.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f66b1ec1640 1 -- 172.21.15.18:0/4220618169 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ac0abf10 con 0x7f66ac13f710 2024-03-25T18:06:22.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f66b1ec1640 10 client.4666.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:22.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f66b1ec1640 10 monclient: _renew_subs 2024-03-25T18:06:22.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f66b1ec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:22.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f66b1ec1640 1 -- 172.21.15.18:0/4220618169 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ac139370 con 0x7f66ac13f710 2024-03-25T18:06:22.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.630+0000 7f66b1ec1640 1 librados: init done 2024-03-25T18:06:22.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.631+0000 7f66b1ec1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:22.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.631+0000 7f66b1ec1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:22.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.631+0000 7f66b1ec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:22.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.631+0000 7f66b1ec1640 1 -- 172.21.15.18:0/4220618169 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ac0627b0 con 0x7f66ac13f710 2024-03-25T18:06:22.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.854+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66a0001d20 con 0x7f66ac13f710 2024-03-25T18:06:22.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.854+0000 7f6693fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:22.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.854+0000 7f66a9ffb640 4 set_mon_vals no callback set 2024-03-25T18:06:22.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.920+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f66a0004d90 con 0x7f66ac13f710 2024-03-25T18:06:22.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.920+0000 7f6693fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:22.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.920+0000 7f6693fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:22.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.920+0000 7f6693fff640 10 monclient: dump: 2024-03-25T18:06:22.921 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:22.921 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:22.921 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:22.921 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:22.921 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:22.921 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:22.921 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:22.921 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:22.921 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:22.921 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:22.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.921+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f66a001ad60 con 0x7f66ac13f710 2024-03-25T18:06:22.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.921+0000 7f66ab7fe640 10 monclient: get_auth_request con 0x7f669806e110 auth_method 0 2024-03-25T18:06:22.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.922+0000 7f6693fff640 10 client.4666.objecter ms_handle_connect 0x7f669806e110 2024-03-25T18:06:22.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.927+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2645856842 0 0) 0x7f66a00549d0 con 0x7f66ac13f710 2024-03-25T18:06:22.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.927+0000 7f6693fff640 10 client.4666.objecter ms_dispatch 0x7f66ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:22.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.927+0000 7f6693fff640 3 client.4666.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:22.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.927+0000 7f6693fff640 3 client.4666.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:22.928 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.927+0000 7f6693fff640 20 client.4666.objecter dump_active .. 0 homeless 2024-03-25T18:06:22.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.929+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f66a0050310 con 0x7f66ac13f710 2024-03-25T18:06:22.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.929+0000 7f6693fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:22.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:22.929+0000 7f6693fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:23.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.157+0000 7f66b1ec1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:23.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.157+0000 7f66b1ec1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:23.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.157+0000 7f66b1ec1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:23.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.157+0000 7f66b1ec1640 1 -- 172.21.15.18:0/4220618169 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66ac0b7180 con 0x7f66ac13f710 2024-03-25T18:06:23.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.180+0000 7f6693fff640 1 -- 172.21.15.18:0/4220618169 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4164665678) 0x7f66a0052410 con 0x7f66ac13f710 2024-03-25T18:06:23.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.180+0000 7f6693fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:23.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.180+0000 7f6693fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:23.181 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:06:23.181 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T17:56:30.756856+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:56:30.756856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:23.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.183+0000 7f66b1ec1640 10 librados: watch_flush enter 2024-03-25T18:06:23.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.183+0000 7f66b1ec1640 10 librados: watch_flush exit 2024-03-25T18:06:23.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.183+0000 7f66b1ec1640 20 client.4666.objecter shutdown clearing up homeless session... 2024-03-25T18:06:23.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.183+0000 7f66b1ec1640 10 client.4666.objecter successfully canceled tick 2024-03-25T18:06:23.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.183+0000 7f66b1ec1640 1 -- 172.21.15.18:0/4220618169 >> v1:172.21.15.18:6800/3721244451 conn(0x7f669806e110 legacy=0x7f66980705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:23.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.183+0000 7f66b1ec1640 10 monclient: shutdown 2024-03-25T18:06:23.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.183+0000 7f66b1ec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:23.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.183+0000 7f66b1ec1640 1 -- 172.21.15.18:0/4220618169 >> v1:172.21.15.169:6789/0 conn(0x7f66ac13f710 legacy=0x7f66ac133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:23.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.183+0000 7f66b1ec1640 1 -- 172.21.15.18:0/4220618169 shutdown_connections 2024-03-25T18:06:23.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.183+0000 7f66b1ec1640 1 -- 172.21.15.18:0/4220618169 >> 172.21.15.18:0/4220618169 conn(0x7f66ac0b7ea0 msgr2=0x7f66ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:23.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.183+0000 7f66b1ec1640 1 -- 172.21.15.18:0/4220618169 shutdown_connections 2024-03-25T18:06:23.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.183+0000 7f66b1ec1640 1 -- 172.21.15.18:0/4220618169 wait complete. 2024-03-25T18:06:23.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:23.183+0000 7f66b1ec1640 1 librados: shutdown 2024-03-25T18:06:23.194 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:29.250 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:27.042 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:29.304 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:27.096 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:29.342 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:27.135 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:29.346 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:27.094 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:29.362 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:27.154 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:29.368 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:27.161 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:29.369 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:27.118 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:29.392 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:27.141 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:29.393 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:27.186 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:30.000 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:30.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.117+0000 7fa7bd820640 10 monclient: get_monmap_and_config 2024-03-25T18:06:30.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.117+0000 7fa7bd820640 10 monclient: build_initial_monmap 2024-03-25T18:06:30.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.117+0000 7fa7bd820640 10 monclient: monmap: 2024-03-25T18:06:30.118 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:06:30.118 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:30.118 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:06:30.117658+0000 2024-03-25T18:06:30.118 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:06:30.117658+0000 2024-03-25T18:06:30.118 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:30.118 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:30.118 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:06:30.118 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:06:30.118 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:06:30.118 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:30.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.117+0000 7fa7bd820640 1 Processor -- start 2024-03-25T18:06:30.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.117+0000 7fa7bd820640 1 -- start start 2024-03-25T18:06:30.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.117+0000 7fa7bd820640 10 monclient: init 2024-03-25T18:06:30.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.118+0000 7fa7bd820640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:30.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.118+0000 7fa7bd820640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:06:30.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.118+0000 7fa7bd820640 10 monclient(hunting): picked mon.noname-b con 0x7fa7b80a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:06:30.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.118+0000 7fa7bd820640 10 monclient(hunting): picked mon.noname-a con 0x7fa7b813f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:06:30.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.118+0000 7fa7bd820640 10 monclient(hunting): picked mon.noname-c con 0x7fa7b81430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:06:30.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.118+0000 7fa7bd820640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b8142520 con 0x7fa7b813f710 2024-03-25T18:06:30.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.118+0000 7fa7bd820640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b80b28a0 con 0x7fa7b81430c0 2024-03-25T18:06:30.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.118+0000 7fa7bd820640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b80b2a30 con 0x7fa7b80a98f0 2024-03-25T18:06:30.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.118+0000 7fa7bd820640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:30.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.118+0000 7fa7bd820640 10 monclient(hunting): authenticate will time out at 7518.696288s 2024-03-25T18:06:30.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.126+0000 7fa7b77fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fa7b81430c0 0x7fa7b814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:53810/0 (socket says 172.21.15.18:53810) 2024-03-25T18:06:30.126 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.126+0000 7fa7b77fe640 1 -- 172.21.15.18:0/116608147 learned_addr learned my addr 172.21.15.18:0/116608147 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:06:30.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.135+0000 7fa7b5ffb640 1 -- 172.21.15.18:0/116608147 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48316089 0 0) 0x7fa7b80b28a0 con 0x7fa7b81430c0 2024-03-25T18:06:30.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.135+0000 7fa7b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:06:30.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.135+0000 7fa7b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:30.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.135+0000 7fa7b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:30.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.135+0000 7fa7b5ffb640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:06:30.135 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.135+0000 7fa7b5ffb640 1 -- 172.21.15.18:0/116608147 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a0003520 con 0x7fa7b81430c0 2024-03-25T18:06:30.140 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.140+0000 7fa7b5ffb640 1 -- 172.21.15.18:0/116608147 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042941482 0 0) 0x7fa7b80b2a30 con 0x7fa7b80a98f0 2024-03-25T18:06:30.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.140+0000 7fa7b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:06:30.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.140+0000 7fa7b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:30.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.140+0000 7fa7b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:30.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.140+0000 7fa7b5ffb640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:06:30.141 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.140+0000 7fa7b5ffb640 1 -- 172.21.15.18:0/116608147 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7b80b28a0 con 0x7fa7b80a98f0 2024-03-25T18:06:30.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7b5ffb640 1 -- 172.21.15.18:0/116608147 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2576678574 0 0) 0x7fa7a0003520 con 0x7fa7b81430c0 2024-03-25T18:06:30.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7b5ffb640 1 -- 172.21.15.18:0/116608147 >> v1:172.21.15.18:6790/0 conn(0x7fa7b80a98f0 legacy=0x7fa7b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:30.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7b5ffb640 1 -- 172.21.15.18:0/116608147 >> v1:172.21.15.18:6789/0 conn(0x7fa7b813f710 legacy=0x7fa7b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:30.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7b5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:06:30.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7b5ffb640 1 monclient: found mon.noname-c 2024-03-25T18:06:30.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:30.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7b5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:06:30.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7b5ffb640 1 -- 172.21.15.18:0/116608147 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b80b44c0 con 0x7fa7b81430c0 2024-03-25T18:06:30.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7b5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:06:30.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7b5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:06:30.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:30.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7b5ffb640 1 -- 172.21.15.18:0/116608147 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa7b0001a10 con 0x7fa7b81430c0 2024-03-25T18:06:30.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7bd820640 5 monclient: authenticate success, global_id 4669 2024-03-25T18:06:30.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7bd820640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:30.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:30.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7b5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:06:30.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7b5ffb640 10 monclient: dump: 2024-03-25T18:06:30.146 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:30.146 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:30.146 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:30.146 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:30.146 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:30.146 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:30.146 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:30.146 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:30.146 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:30.146 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:30.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.144+0000 7fa7bd820640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:30.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.154+0000 7fa7b5ffb640 1 -- 172.21.15.18:0/116608147 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7b0001b70 con 0x7fa7b81430c0 2024-03-25T18:06:30.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.154+0000 7fa7b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:30.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.154+0000 7fa7b5ffb640 1 -- 172.21.15.18:0/116608147 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa7b0001f80 con 0x7fa7b81430c0 2024-03-25T18:06:30.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.154+0000 7fa7b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:30.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.154+0000 7fa7b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:30.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.154+0000 7fa7b5ffb640 10 monclient: dump: 2024-03-25T18:06:30.155 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:30.155 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:30.155 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:30.155 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:30.155 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:30.155 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:30.155 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:30.155 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:30.155 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:30.155 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:30.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.154+0000 7fa7bd820640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:30.155 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.154+0000 7fa7bd820640 4 set_mon_vals no callback set 2024-03-25T18:06:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.155+0000 7fa7bd820640 10 monclient: shutdown 2024-03-25T18:06:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.155+0000 7fa7bd820640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.155+0000 7fa7bd820640 1 -- 172.21.15.18:0/116608147 >> v1:172.21.15.169:6789/0 conn(0x7fa7b81430c0 legacy=0x7fa7b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.155+0000 7fa7bd820640 1 -- 172.21.15.18:0/116608147 shutdown_connections 2024-03-25T18:06:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.155+0000 7fa7bd820640 1 -- 172.21.15.18:0/116608147 >> 172.21.15.18:0/116608147 conn(0x7fa7b80b7ea0 msgr2=0x7fa7b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.155+0000 7fa7bd820640 1 -- 172.21.15.18:0/116608147 shutdown_connections 2024-03-25T18:06:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.155+0000 7fa7bd820640 1 -- 172.21.15.18:0/116608147 wait complete. 2024-03-25T18:06:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.156+0000 7fa7bd820640 10 monclient: build_initial_monmap 2024-03-25T18:06:30.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.156+0000 7fa7bd820640 10 monclient: monmap: 2024-03-25T18:06:30.157 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:06:30.157 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:30.158 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:06:30.158 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:06:30.158 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:30.158 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:06:30.158 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:06:30.158 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:06:30.158 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:06:30.158 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:30.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.156+0000 7fa7bd820640 1 librados: starting msgr at 2024-03-25T18:06:30.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.156+0000 7fa7bd820640 1 librados: starting objecter 2024-03-25T18:06:30.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.156+0000 7fa7bd820640 1 Processor -- start 2024-03-25T18:06:30.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.156+0000 7fa7bd820640 1 -- start start 2024-03-25T18:06:30.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.156+0000 7fa7bd820640 1 librados: setting wanted keys 2024-03-25T18:06:30.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.156+0000 7fa7bd820640 1 librados: calling monclient init 2024-03-25T18:06:30.158 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.156+0000 7fa7bd820640 10 monclient: init 2024-03-25T18:06:30.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.157+0000 7fa7bd820640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:30.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.157+0000 7fa7bd820640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:06:30.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.157+0000 7fa7bd820640 10 monclient(hunting): picked mon.noname-c con 0x7fa7b80a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:06:30.159 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.157+0000 7fa7bd820640 10 monclient(hunting): picked mon.noname-b con 0x7fa7b813f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:06:30.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.157+0000 7fa7bd820640 10 monclient(hunting): picked mon.noname-a con 0x7fa7b81430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:06:30.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.157+0000 7fa7bd820640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b8136ec0 con 0x7fa7b81430c0 2024-03-25T18:06:30.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.157+0000 7fa7bd820640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b8136c20 con 0x7fa7b813f710 2024-03-25T18:06:30.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.157+0000 7fa7bd820640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b8137db0 con 0x7fa7b80a98f0 2024-03-25T18:06:30.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.157+0000 7fa7bd820640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:30.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.157+0000 7fa7bd820640 10 monclient(hunting): authenticate will time out at 7518.735577s 2024-03-25T18:06:30.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.169+0000 7fa7b6ffd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fa7b80a98f0 0x7fa7b81376a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:56958/0 (socket says 172.21.15.18:56958) 2024-03-25T18:06:30.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.169+0000 7fa7b6ffd640 1 -- 172.21.15.18:0/1468992416 learned_addr learned my addr 172.21.15.18:0/1468992416 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:06:30.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.170+0000 7fa79ffff640 10 client.?.objecter ms_handle_connect 0x7fa7b81430c0 2024-03-25T18:06:30.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.170+0000 7fa79ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:30.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.170+0000 7fa79ffff640 10 client.?.objecter ms_handle_connect 0x7fa7b80a98f0 2024-03-25T18:06:30.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.170+0000 7fa79ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:30.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.173+0000 7fa79ffff640 10 client.?.objecter ms_handle_connect 0x7fa7b813f710 2024-03-25T18:06:30.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.173+0000 7fa79ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:30.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.187+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825719812 0 0) 0x7fa7b8136ec0 con 0x7fa7b81430c0 2024-03-25T18:06:30.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.187+0000 7fa79ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:30.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.187+0000 7fa79ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:30.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.187+0000 7fa79ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:30.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.187+0000 7fa79ffff640 10 monclient(hunting): my global_id is 4683 2024-03-25T18:06:30.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.187+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a4004eb0 con 0x7fa7b81430c0 2024-03-25T18:06:30.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.188+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957410663 0 0) 0x7fa7b8137db0 con 0x7fa7b80a98f0 2024-03-25T18:06:30.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.188+0000 7fa79ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:30.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.188+0000 7fa79ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:30.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.188+0000 7fa79ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:30.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.188+0000 7fa79ffff640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:06:30.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.188+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7b8136ec0 con 0x7fa7b80a98f0 2024-03-25T18:06:30.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.190+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559521773 0 0) 0x7fa7b8136c20 con 0x7fa7b813f710 2024-03-25T18:06:30.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.190+0000 7fa79ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:30.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.190+0000 7fa79ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:30.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.190+0000 7fa79ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:30.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.190+0000 7fa79ffff640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:06:30.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.190+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7b8137db0 con 0x7fa7b813f710 2024-03-25T18:06:30.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.202+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1720001321 0 0) 0x7fa7b8137db0 con 0x7fa7b813f710 2024-03-25T18:06:30.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.202+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7b8136c20 con 0x7fa7b813f710 2024-03-25T18:06:30.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.202+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa7ac001860 con 0x7fa7b813f710 2024-03-25T18:06:30.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.202+0000 7fa79ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:30.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.202+0000 7fa79ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:30.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.202+0000 7fa79ffff640 10 monclient(hunting): dump: 2024-03-25T18:06:30.203 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:30.203 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:30.204 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:30.204 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:30.204 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:30.204 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:30.204 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:30.204 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:30.204 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:30.204 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:30.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.202+0000 7fa79ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:30.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.212+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 683922044 0 0) 0x7fa7a4004eb0 con 0x7fa7b81430c0 2024-03-25T18:06:30.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.212+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7b8137db0 con 0x7fa7b81430c0 2024-03-25T18:06:30.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.212+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa7b0001ec0 con 0x7fa7b81430c0 2024-03-25T18:06:30.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.212+0000 7fa79ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:30.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.212+0000 7fa79ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:06:30.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.212+0000 7fa79ffff640 10 monclient(hunting): dump: 2024-03-25T18:06:30.213 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:30.213 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:30.213 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:30.213 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:30.213 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:30.214 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:30.214 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:30.214 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:30.214 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:30.214 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:30.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 398396745 0 0) 0x7fa7b8136c20 con 0x7fa7b813f710 2024-03-25T18:06:30.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 >> v1:172.21.15.18:6790/0 conn(0x7fa7b80a98f0 legacy=0x7fa7b81376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:30.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 >> v1:172.21.15.18:6789/0 conn(0x7fa7b81430c0 legacy=0x7fa7b81e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:30.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa79ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:06:30.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa79ffff640 1 monclient: found mon.b 2024-03-25T18:06:30.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa79ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:30.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa79ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:30.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b8138b50 con 0x7fa7b813f710 2024-03-25T18:06:30.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa79ffff640 10 monclient: _finish_auth 0 2024-03-25T18:06:30.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa79ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:06:30.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa79ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:30.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa7bd820640 5 monclient: authenticate success, global_id 4672 2024-03-25T18:06:30.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa7bd820640 10 monclient: _renew_subs 2024-03-25T18:06:30.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa7bd820640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:30.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa7bd820640 1 -- 172.21.15.18:0/1468992416 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b8139830 con 0x7fa7b813f710 2024-03-25T18:06:30.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa7bd820640 10 client.4672.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:30.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa7bd820640 10 monclient: _renew_subs 2024-03-25T18:06:30.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa7bd820640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:30.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa7bd820640 1 -- 172.21.15.18:0/1468992416 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b81369b0 con 0x7fa7b813f710 2024-03-25T18:06:30.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.218+0000 7fa7bd820640 1 librados: init done 2024-03-25T18:06:30.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.219+0000 7fa7bd820640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:30.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.219+0000 7fa7bd820640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:30.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.219+0000 7fa7bd820640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:30.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.219+0000 7fa7bd820640 1 -- 172.21.15.18:0/1468992416 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b81373c0 con 0x7fa7b813f710 2024-03-25T18:06:30.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.523+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7ac001d30 con 0x7fa7b813f710 2024-03-25T18:06:30.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.523+0000 7fa79ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:30.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.523+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fa7ac0035f0 con 0x7fa7b813f710 2024-03-25T18:06:30.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.523+0000 7fa79ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:30.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.523+0000 7fa79ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:30.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.523+0000 7fa79ffff640 10 monclient: dump: 2024-03-25T18:06:30.524 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:30.524 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:30.524 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:30.524 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:30.524 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:30.524 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:30.524 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:30.524 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:30.524 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:30.524 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:30.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.523+0000 7fa7b57fa640 4 set_mon_vals no callback set 2024-03-25T18:06:30.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.524+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fa7ac019640 con 0x7fa7b813f710 2024-03-25T18:06:30.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.524+0000 7fa7b6ffd640 10 monclient: get_auth_request con 0x7fa7a406df70 auth_method 0 2024-03-25T18:06:30.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.524+0000 7fa79ffff640 10 client.4672.objecter ms_handle_connect 0x7fa7a406df70 2024-03-25T18:06:30.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.529+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 865634137 0 0) 0x7fa7ac053380 con 0x7fa7b813f710 2024-03-25T18:06:30.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.529+0000 7fa79ffff640 10 client.4672.objecter ms_dispatch 0x7fa7b8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:06:30.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.529+0000 7fa79ffff640 3 client.4672.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:06:30.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.529+0000 7fa79ffff640 3 client.4672.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:06:30.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.529+0000 7fa79ffff640 20 client.4672.objecter dump_active .. 0 homeless 2024-03-25T18:06:30.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.530+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7ac04ecc0 con 0x7fa7b813f710 2024-03-25T18:06:30.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.530+0000 7fa79ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:30.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.530+0000 7fa79ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:30.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.750+0000 7fa7bd820640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:30.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.750+0000 7fa7bd820640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:30.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.750+0000 7fa7bd820640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:30.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.750+0000 7fa7bd820640 1 -- 172.21.15.18:0/1468992416 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7b814f320 con 0x7fa7b813f710 2024-03-25T18:06:30.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.781+0000 7fa79ffff640 1 -- 172.21.15.18:0/1468992416 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 678331228) 0x7fa7ac050dc0 con 0x7fa7b813f710 2024-03-25T18:06:30.781 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.781+0000 7fa79ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:30.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.781+0000 7fa79ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:30.782 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:06:30.782 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T18:06:29.006918+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:06:29.006918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:30.784 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.783+0000 7fa7bd820640 10 librados: watch_flush enter 2024-03-25T18:06:30.784 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.783+0000 7fa7bd820640 10 librados: watch_flush exit 2024-03-25T18:06:30.784 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.783+0000 7fa7bd820640 20 client.4672.objecter shutdown clearing up homeless session... 2024-03-25T18:06:30.785 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.783+0000 7fa7bd820640 10 client.4672.objecter successfully canceled tick 2024-03-25T18:06:30.785 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.784+0000 7fa7bd820640 1 -- 172.21.15.18:0/1468992416 >> v1:172.21.15.18:6800/3721244451 conn(0x7fa7a406df70 legacy=0x7fa7a4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:30.785 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.784+0000 7fa7bd820640 10 monclient: shutdown 2024-03-25T18:06:30.785 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.784+0000 7fa7bd820640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:30.785 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.784+0000 7fa7bd820640 1 -- 172.21.15.18:0/1468992416 >> v1:172.21.15.169:6789/0 conn(0x7fa7b813f710 legacy=0x7fa7b80b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:30.785 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.784+0000 7fa7bd820640 1 -- 172.21.15.18:0/1468992416 shutdown_connections 2024-03-25T18:06:30.785 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.784+0000 7fa7bd820640 1 -- 172.21.15.18:0/1468992416 >> 172.21.15.18:0/1468992416 conn(0x7fa7b80b7ea0 msgr2=0x7fa7b813ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:30.785 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.784+0000 7fa7bd820640 1 -- 172.21.15.18:0/1468992416 shutdown_connections 2024-03-25T18:06:30.785 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.784+0000 7fa7bd820640 1 -- 172.21.15.18:0/1468992416 wait complete. 2024-03-25T18:06:30.785 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:30.784+0000 7fa7bd820640 1 librados: shutdown 2024-03-25T18:06:30.796 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:37.603 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:37.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.720+0000 7ff1f7d9d640 10 monclient: get_monmap_and_config 2024-03-25T18:06:37.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.720+0000 7ff1f7d9d640 10 monclient: build_initial_monmap 2024-03-25T18:06:37.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.720+0000 7ff1f7d9d640 10 monclient: monmap: 2024-03-25T18:06:37.721 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:06:37.721 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:37.721 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:06:37.721049+0000 2024-03-25T18:06:37.721 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:06:37.721049+0000 2024-03-25T18:06:37.721 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:37.721 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:37.722 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:06:37.722 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:06:37.722 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:06:37.722 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:37.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.721+0000 7ff1f7d9d640 1 Processor -- start 2024-03-25T18:06:37.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.721+0000 7ff1f7d9d640 1 -- start start 2024-03-25T18:06:37.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.721+0000 7ff1f7d9d640 10 monclient: init 2024-03-25T18:06:37.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.722+0000 7ff1f7d9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:37.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.722+0000 7ff1f7d9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:06:37.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.722+0000 7ff1f7d9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff1f00a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:06:37.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.722+0000 7ff1f7d9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff1f013f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:06:37.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.722+0000 7ff1f7d9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff1f01430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:06:37.723 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.722+0000 7ff1f7d9d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f0142520 con 0x7ff1f00a98f0 2024-03-25T18:06:37.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.722+0000 7ff1f7d9d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f00b28a0 con 0x7ff1f013f710 2024-03-25T18:06:37.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.722+0000 7ff1f7d9d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f00b2a30 con 0x7ff1f01430c0 2024-03-25T18:06:37.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.722+0000 7ff1f7d9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:37.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.722+0000 7ff1f7d9d640 10 monclient(hunting): authenticate will time out at 7526.300306s 2024-03-25T18:06:37.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.727+0000 7ff1f5338640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7ff1f013f710 0x7ff1f01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:39768/0 (socket says 172.21.15.18:39768) 2024-03-25T18:06:37.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.727+0000 7ff1f5338640 1 -- 172.21.15.18:0/687084759 learned_addr learned my addr 172.21.15.18:0/687084759 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:06:37.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.741+0000 7ff1f4b37640 1 -- 172.21.15.18:0/687084759 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920538323 0 0) 0x7ff1f0142520 con 0x7ff1f00a98f0 2024-03-25T18:06:37.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.741+0000 7ff1f4b37640 10 monclient(hunting): _negotiate 2024-03-25T18:06:37.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.741+0000 7ff1f4b37640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:37.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.741+0000 7ff1f4b37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:37.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.741+0000 7ff1f4b37640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:06:37.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.741+0000 7ff1f4b37640 1 -- 172.21.15.18:0/687084759 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1dc003520 con 0x7ff1f00a98f0 2024-03-25T18:06:37.752 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.752+0000 7ff1f4b37640 1 -- 172.21.15.18:0/687084759 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904423545 0 0) 0x7ff1f00b28a0 con 0x7ff1f013f710 2024-03-25T18:06:37.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.752+0000 7ff1f4b37640 10 monclient(hunting): _negotiate 2024-03-25T18:06:37.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.752+0000 7ff1f4b37640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:37.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.752+0000 7ff1f4b37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:37.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.752+0000 7ff1f4b37640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:06:37.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.752+0000 7ff1f4b37640 1 -- 172.21.15.18:0/687084759 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f0142520 con 0x7ff1f013f710 2024-03-25T18:06:37.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.753+0000 7ff1f4b37640 1 -- 172.21.15.18:0/687084759 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563958953 0 0) 0x7ff1f00b2a30 con 0x7ff1f01430c0 2024-03-25T18:06:37.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.753+0000 7ff1f4b37640 10 monclient(hunting): _negotiate 2024-03-25T18:06:37.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.753+0000 7ff1f4b37640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:37.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.753+0000 7ff1f4b37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:37.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.753+0000 7ff1f4b37640 10 monclient(hunting): my global_id is 4694 2024-03-25T18:06:37.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.753+0000 7ff1f4b37640 1 -- 172.21.15.18:0/687084759 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f00b28a0 con 0x7ff1f01430c0 2024-03-25T18:06:37.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.760+0000 7ff1f4b37640 1 -- 172.21.15.18:0/687084759 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2188586515 0 0) 0x7ff1f0142520 con 0x7ff1f013f710 2024-03-25T18:06:37.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.760+0000 7ff1f4b37640 1 -- 172.21.15.18:0/687084759 >> v1:172.21.15.18:6790/0 conn(0x7ff1f01430c0 legacy=0x7ff1f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:37.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.760+0000 7ff1f4b37640 1 -- 172.21.15.18:0/687084759 >> v1:172.21.15.18:6789/0 conn(0x7ff1f00a98f0 legacy=0x7ff1f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:37.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.760+0000 7ff1f4b37640 10 monclient: _finish_hunting 0 2024-03-25T18:06:37.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.760+0000 7ff1f4b37640 1 monclient: found mon.noname-c 2024-03-25T18:06:37.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.760+0000 7ff1f4b37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:37.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.760+0000 7ff1f4b37640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:06:37.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.760+0000 7ff1f4b37640 1 -- 172.21.15.18:0/687084759 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f00b44c0 con 0x7ff1f013f710 2024-03-25T18:06:37.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.760+0000 7ff1f4b37640 10 monclient: _finish_auth 0 2024-03-25T18:06:37.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.760+0000 7ff1f4b37640 10 monclient: _check_auth_tickets 2024-03-25T18:06:37.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.760+0000 7ff1f4b37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:37.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.760+0000 7ff1f4b37640 1 -- 172.21.15.18:0/687084759 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ff1e0001670 con 0x7ff1f013f710 2024-03-25T18:06:37.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.760+0000 7ff1f4b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:37.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.760+0000 7ff1f4b37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:06:37.762 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.760+0000 7ff1f4b37640 10 monclient: dump: 2024-03-25T18:06:37.762 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:37.762 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:37.762 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:37.762 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:37.762 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:37.762 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:37.762 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:37.762 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:37.762 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:37.763 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:37.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.760+0000 7ff1f7d9d640 5 monclient: authenticate success, global_id 4675 2024-03-25T18:06:37.763 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.760+0000 7ff1f7d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:37.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.782+0000 7ff1f4b37640 1 -- 172.21.15.18:0/687084759 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1e00017d0 con 0x7ff1f013f710 2024-03-25T18:06:37.782 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.782+0000 7ff1f4b37640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:37.783 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.782+0000 7ff1f4b37640 1 -- 172.21.15.18:0/687084759 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ff1e0001be0 con 0x7ff1f013f710 2024-03-25T18:06:37.783 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.782+0000 7ff1f7d9d640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:37.783 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.782+0000 7ff1f7d9d640 4 set_mon_vals no callback set 2024-03-25T18:06:37.783 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.782+0000 7ff1f4b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:37.783 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.782+0000 7ff1f4b37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:37.783 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.782+0000 7ff1f4b37640 10 monclient: dump: 2024-03-25T18:06:37.783 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:37.783 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:37.783 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:37.783 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:37.783 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:37.783 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:37.784 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:37.784 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:37.784 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:37.784 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:37.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.783+0000 7ff1f7d9d640 10 monclient: shutdown 2024-03-25T18:06:37.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.783+0000 7ff1f7d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:37.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.783+0000 7ff1f7d9d640 1 -- 172.21.15.18:0/687084759 >> v1:172.21.15.169:6789/0 conn(0x7ff1f013f710 legacy=0x7ff1f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:37.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.783+0000 7ff1f7d9d640 1 -- 172.21.15.18:0/687084759 shutdown_connections 2024-03-25T18:06:37.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.783+0000 7ff1f7d9d640 1 -- 172.21.15.18:0/687084759 >> 172.21.15.18:0/687084759 conn(0x7ff1f00b7ea0 msgr2=0x7ff1f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:37.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.783+0000 7ff1f7d9d640 1 -- 172.21.15.18:0/687084759 shutdown_connections 2024-03-25T18:06:37.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.783+0000 7ff1f7d9d640 1 -- 172.21.15.18:0/687084759 wait complete. 2024-03-25T18:06:37.786 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.784+0000 7ff1f7d9d640 10 monclient: build_initial_monmap 2024-03-25T18:06:37.787 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.784+0000 7ff1f7d9d640 10 monclient: monmap: 2024-03-25T18:06:37.787 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:06:37.787 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:37.787 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:06:37.787 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:06:37.787 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:37.787 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:06:37.787 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:06:37.787 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:06:37.787 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:06:37.787 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:37.787 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.784+0000 7ff1f7d9d640 1 librados: starting msgr at 2024-03-25T18:06:37.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.784+0000 7ff1f7d9d640 1 librados: starting objecter 2024-03-25T18:06:37.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.784+0000 7ff1f7d9d640 1 Processor -- start 2024-03-25T18:06:37.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.784+0000 7ff1f7d9d640 1 -- start start 2024-03-25T18:06:37.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.784+0000 7ff1f7d9d640 1 librados: setting wanted keys 2024-03-25T18:06:37.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.784+0000 7ff1f7d9d640 1 librados: calling monclient init 2024-03-25T18:06:37.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.784+0000 7ff1f7d9d640 10 monclient: init 2024-03-25T18:06:37.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.785+0000 7ff1f7d9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:37.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.785+0000 7ff1f7d9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:06:37.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.785+0000 7ff1f7d9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff1f00a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:06:37.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.785+0000 7ff1f7d9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff1f013f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:06:37.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.785+0000 7ff1f7d9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff1f01430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:06:37.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.785+0000 7ff1f7d9d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f0136510 con 0x7ff1f00a98f0 2024-03-25T18:06:37.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.785+0000 7ff1f7d9d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f00abb00 con 0x7ff1f013f710 2024-03-25T18:06:37.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.785+0000 7ff1f7d9d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f01da990 con 0x7ff1f01430c0 2024-03-25T18:06:37.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.785+0000 7ff1f7d9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:37.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.785+0000 7ff1f7d9d640 10 monclient(hunting): authenticate will time out at 7526.363251s 2024-03-25T18:06:37.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.794+0000 7ff1f5338640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7ff1f013f710 0x7ff1f01383a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:39818/0 (socket says 172.21.15.18:39818) 2024-03-25T18:06:37.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.794+0000 7ff1f5338640 1 -- 172.21.15.18:0/1978831341 learned_addr learned my addr 172.21.15.18:0/1978831341 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:06:37.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.797+0000 7ff1e67fc640 10 client.?.objecter ms_handle_connect 0x7ff1f01430c0 2024-03-25T18:06:37.797 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.797+0000 7ff1e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:37.798 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.798+0000 7ff1e67fc640 10 client.?.objecter ms_handle_connect 0x7ff1f013f710 2024-03-25T18:06:37.798 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.798+0000 7ff1e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:37.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.799+0000 7ff1e67fc640 10 client.?.objecter ms_handle_connect 0x7ff1f00a98f0 2024-03-25T18:06:37.799 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.799+0000 7ff1e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:37.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.804+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295140534 0 0) 0x7ff1f01da990 con 0x7ff1f01430c0 2024-03-25T18:06:37.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.804+0000 7ff1e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:37.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.804+0000 7ff1e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:37.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.804+0000 7ff1e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:37.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.804+0000 7ff1e67fc640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:06:37.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.804+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1d4004eb0 con 0x7ff1f01430c0 2024-03-25T18:06:37.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.824+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280541668 0 0) 0x7ff1f00abb00 con 0x7ff1f013f710 2024-03-25T18:06:37.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.824+0000 7ff1e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:37.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.824+0000 7ff1e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:37.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.824+0000 7ff1e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:37.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.824+0000 7ff1e67fc640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:06:37.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.824+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f01da990 con 0x7ff1f013f710 2024-03-25T18:06:37.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.825+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3518916727 0 0) 0x7ff1d4004eb0 con 0x7ff1f01430c0 2024-03-25T18:06:37.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.825+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1f00abb00 con 0x7ff1f01430c0 2024-03-25T18:06:37.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.825+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ff1e8001bd0 con 0x7ff1f01430c0 2024-03-25T18:06:37.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.825+0000 7ff1e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:37.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.825+0000 7ff1e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:06:37.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.825+0000 7ff1e67fc640 10 monclient(hunting): dump: 2024-03-25T18:06:37.827 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:37.827 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:37.827 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:37.827 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:37.827 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:37.827 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:37.827 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:37.827 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:37.827 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:37.827 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:37.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.826+0000 7ff1e67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:37.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.827+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491903669 0 0) 0x7ff1f0136510 con 0x7ff1f00a98f0 2024-03-25T18:06:37.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.827+0000 7ff1e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:37.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.827+0000 7ff1e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:37.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.827+0000 7ff1e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:37.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.827+0000 7ff1e67fc640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:06:37.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.827+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1d4004eb0 con 0x7ff1f00a98f0 2024-03-25T18:06:37.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.845+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 166675896 0 0) 0x7ff1f01da990 con 0x7ff1f013f710 2024-03-25T18:06:37.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.845+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1f0136510 con 0x7ff1f013f710 2024-03-25T18:06:37.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.845+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ff1e0001ba0 con 0x7ff1f013f710 2024-03-25T18:06:37.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.845+0000 7ff1e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:37.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.845+0000 7ff1e67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:37.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.845+0000 7ff1e67fc640 10 monclient(hunting): dump: 2024-03-25T18:06:37.846 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:37.846 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:37.846 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:37.846 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:37.846 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:37.846 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:37.847 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:37.847 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:37.847 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:37.847 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:37.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.861+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1219274274 0 0) 0x7ff1f0136510 con 0x7ff1f013f710 2024-03-25T18:06:37.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.861+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 >> v1:172.21.15.18:6790/0 conn(0x7ff1f01430c0 legacy=0x7ff1f01e1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:37.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.861+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 >> v1:172.21.15.18:6789/0 conn(0x7ff1f00a98f0 legacy=0x7ff1f01daeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:37.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.861+0000 7ff1e67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:06:37.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.861+0000 7ff1e67fc640 1 monclient: found mon.b 2024-03-25T18:06:37.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.861+0000 7ff1e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:37.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.861+0000 7ff1e67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:37.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.861+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f0137820 con 0x7ff1f013f710 2024-03-25T18:06:37.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.861+0000 7ff1e67fc640 10 monclient: _finish_auth 0 2024-03-25T18:06:37.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.861+0000 7ff1e67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:06:37.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.861+0000 7ff1e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:37.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.861+0000 7ff1f7d9d640 5 monclient: authenticate success, global_id 4678 2024-03-25T18:06:37.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.862+0000 7ff1f7d9d640 10 monclient: _renew_subs 2024-03-25T18:06:37.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.862+0000 7ff1f7d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:37.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.862+0000 7ff1f7d9d640 1 -- 172.21.15.18:0/1978831341 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f01db5c0 con 0x7ff1f013f710 2024-03-25T18:06:37.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.862+0000 7ff1f7d9d640 10 client.4678.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:37.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.862+0000 7ff1f7d9d640 10 monclient: _renew_subs 2024-03-25T18:06:37.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.862+0000 7ff1f7d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:37.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.862+0000 7ff1f7d9d640 1 -- 172.21.15.18:0/1978831341 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f0138c90 con 0x7ff1f013f710 2024-03-25T18:06:37.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.862+0000 7ff1f7d9d640 1 librados: init done 2024-03-25T18:06:37.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.863+0000 7ff1f7d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:37.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.863+0000 7ff1f7d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:37.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.863+0000 7ff1f7d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:37.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:37.863+0000 7ff1f7d9d640 1 -- 172.21.15.18:0/1978831341 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f00b3ef0 con 0x7ff1f013f710 2024-03-25T18:06:38.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.171+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1e0004a90 con 0x7ff1f013f710 2024-03-25T18:06:38.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.171+0000 7ff1e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:38.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.171+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7ff1e0004da0 con 0x7ff1f013f710 2024-03-25T18:06:38.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.171+0000 7ff1e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:38.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.171+0000 7ff1e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:38.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.171+0000 7ff1e67fc640 10 monclient: dump: 2024-03-25T18:06:38.172 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:38.172 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:38.172 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:38.172 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:38.172 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:38.172 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:38.173 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:38.173 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:38.173 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:38.173 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:38.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.171+0000 7ff1e7fff640 4 set_mon_vals no callback set 2024-03-25T18:06:38.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.172+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7ff1e001ad70 con 0x7ff1f013f710 2024-03-25T18:06:38.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.173+0000 7ff1f5b39640 10 monclient: get_auth_request con 0x7ff1d406ded0 auth_method 0 2024-03-25T18:06:38.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.173+0000 7ff1e67fc640 10 client.4678.objecter ms_handle_connect 0x7ff1d406ded0 2024-03-25T18:06:38.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.177+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 865634137 0 0) 0x7ff1e0054940 con 0x7ff1f013f710 2024-03-25T18:06:38.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.178+0000 7ff1e67fc640 10 client.4678.objecter ms_dispatch 0x7ff1f0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:06:38.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.178+0000 7ff1e67fc640 3 client.4678.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:06:38.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.178+0000 7ff1e67fc640 3 client.4678.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:06:38.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.178+0000 7ff1e67fc640 20 client.4678.objecter dump_active .. 0 homeless 2024-03-25T18:06:38.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.178+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1e0050280 con 0x7ff1f013f710 2024-03-25T18:06:38.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.178+0000 7ff1e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:38.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.178+0000 7ff1e67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:38.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.449+0000 7ff1f7d9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:38.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.449+0000 7ff1f7d9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:38.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.449+0000 7ff1f7d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:06:38.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.449+0000 7ff1f7d9d640 1 -- 172.21.15.18:0/1978831341 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1f014f320 con 0x7ff1f013f710 2024-03-25T18:06:38.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.499+0000 7ff1e67fc640 1 -- 172.21.15.18:0/1978831341 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 678331228) 0x7ff1e0052380 con 0x7ff1f013f710 2024-03-25T18:06:38.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.499+0000 7ff1e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:38.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.499+0000 7ff1e67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:38.499 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:06:38.500 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T18:06:29.006918+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:06:29.006918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:38.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.501+0000 7ff1f7d9d640 10 librados: watch_flush enter 2024-03-25T18:06:38.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.501+0000 7ff1f7d9d640 10 librados: watch_flush exit 2024-03-25T18:06:38.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.501+0000 7ff1f7d9d640 20 client.4678.objecter shutdown clearing up homeless session... 2024-03-25T18:06:38.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.501+0000 7ff1f7d9d640 10 client.4678.objecter successfully canceled tick 2024-03-25T18:06:38.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.501+0000 7ff1f7d9d640 1 -- 172.21.15.18:0/1978831341 >> v1:172.21.15.18:6800/3721244451 conn(0x7ff1d406ded0 legacy=0x7ff1d4070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:38.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.501+0000 7ff1f7d9d640 10 monclient: shutdown 2024-03-25T18:06:38.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.501+0000 7ff1f7d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:38.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.501+0000 7ff1f7d9d640 1 -- 172.21.15.18:0/1978831341 >> v1:172.21.15.169:6789/0 conn(0x7ff1f013f710 legacy=0x7ff1f01383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:38.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.502+0000 7ff1f7d9d640 1 -- 172.21.15.18:0/1978831341 shutdown_connections 2024-03-25T18:06:38.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.502+0000 7ff1f7d9d640 1 -- 172.21.15.18:0/1978831341 >> 172.21.15.18:0/1978831341 conn(0x7ff1f00b7ea0 msgr2=0x7ff1f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:38.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.502+0000 7ff1f7d9d640 1 -- 172.21.15.18:0/1978831341 shutdown_connections 2024-03-25T18:06:38.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.502+0000 7ff1f7d9d640 1 -- 172.21.15.18:0/1978831341 wait complete. 2024-03-25T18:06:38.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:38.502+0000 7ff1f7d9d640 1 librados: shutdown 2024-03-25T18:06:38.514 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:45.320 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:45.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.433+0000 7f1ea5c40640 10 monclient: get_monmap_and_config 2024-03-25T18:06:45.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.433+0000 7f1ea5c40640 10 monclient: build_initial_monmap 2024-03-25T18:06:45.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.433+0000 7f1ea5c40640 10 monclient: monmap: 2024-03-25T18:06:45.434 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:06:45.434 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:45.434 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:06:45.434436+0000 2024-03-25T18:06:45.435 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:06:45.434436+0000 2024-03-25T18:06:45.435 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:45.435 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:45.435 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:06:45.435 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:06:45.435 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:06:45.435 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:45.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.434+0000 7f1ea5c40640 1 Processor -- start 2024-03-25T18:06:45.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.434+0000 7f1ea5c40640 1 -- start start 2024-03-25T18:06:45.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.434+0000 7f1ea5c40640 10 monclient: init 2024-03-25T18:06:45.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.434+0000 7f1ea5c40640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:45.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.434+0000 7f1ea5c40640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:06:45.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.434+0000 7f1ea5c40640 10 monclient(hunting): picked mon.noname-c con 0x7f1ea00b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T18:06:45.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.434+0000 7f1ea5c40640 10 monclient(hunting): picked mon.noname-b con 0x7f1ea00b1730 addr v1:172.21.15.18:6790/0 2024-03-25T18:06:45.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.434+0000 7f1ea5c40640 10 monclient(hunting): picked mon.noname-a con 0x7f1ea014be90 addr v1:172.21.15.18:6789/0 2024-03-25T18:06:45.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.434+0000 7f1ea5c40640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea00af940 con 0x7f1ea014be90 2024-03-25T18:06:45.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.434+0000 7f1ea5c40640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea00b0a70 con 0x7f1ea00b4d60 2024-03-25T18:06:45.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.434+0000 7f1ea5c40640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea00b0d50 con 0x7f1ea00b1730 2024-03-25T18:06:45.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.434+0000 7f1ea5c40640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:45.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.434+0000 7f1ea5c40640 10 monclient(hunting): authenticate will time out at 7534.013472s 2024-03-25T18:06:45.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.455+0000 7f1e9effd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f1ea00b1730 0x7f1ea00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:49906/0 (socket says 172.21.15.18:49906) 2024-03-25T18:06:45.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.455+0000 7f1e9effd640 1 -- 172.21.15.18:0/522069225 learned_addr learned my addr 172.21.15.18:0/522069225 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:06:45.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.461+0000 7f1e9e7fc640 1 -- 172.21.15.18:0/522069225 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684115168 0 0) 0x7f1ea00b0d50 con 0x7f1ea00b1730 2024-03-25T18:06:45.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.461+0000 7f1e9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:45.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.461+0000 7f1e9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:45.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.461+0000 7f1e9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:45.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.461+0000 7f1e9e7fc640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:06:45.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.461+0000 7f1e9e7fc640 1 -- 172.21.15.18:0/522069225 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e88003520 con 0x7f1ea00b1730 2024-03-25T18:06:45.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.468+0000 7f1e9e7fc640 1 -- 172.21.15.18:0/522069225 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2361734780 0 0) 0x7f1e88003520 con 0x7f1ea00b1730 2024-03-25T18:06:45.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.468+0000 7f1e9e7fc640 1 -- 172.21.15.18:0/522069225 >> v1:172.21.15.169:6789/0 conn(0x7f1ea00b4d60 legacy=0x7f1ea013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:45.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.468+0000 7f1e9e7fc640 1 -- 172.21.15.18:0/522069225 >> v1:172.21.15.18:6789/0 conn(0x7f1ea014be90 legacy=0x7f1ea014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:45.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.468+0000 7f1e9e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:06:45.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.468+0000 7f1e9e7fc640 1 monclient: found mon.noname-b 2024-03-25T18:06:45.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.468+0000 7f1e9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:45.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.468+0000 7f1e9e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:06:45.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.468+0000 7f1e9e7fc640 1 -- 172.21.15.18:0/522069225 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea00b1030 con 0x7f1ea00b1730 2024-03-25T18:06:45.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.468+0000 7f1e9e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:06:45.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.468+0000 7f1e9e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:06:45.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.468+0000 7f1e9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:45.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.468+0000 7f1e9e7fc640 1 -- 172.21.15.18:0/522069225 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f1e94001690 con 0x7f1ea00b1730 2024-03-25T18:06:45.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.468+0000 7f1e9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:45.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.468+0000 7f1e9e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:45.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.468+0000 7f1e9e7fc640 10 monclient: dump: 2024-03-25T18:06:45.471 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:45.471 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:45.473 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:45.473 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:45.473 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:45.473 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:45.473 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:45.473 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:45.473 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:45.473 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:45.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.469+0000 7f1ea5c40640 5 monclient: authenticate success, global_id 4700 2024-03-25T18:06:45.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.469+0000 7f1ea5c40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:45.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.475+0000 7f1e9e7fc640 1 -- 172.21.15.18:0/522069225 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e940017f0 con 0x7f1ea00b1730 2024-03-25T18:06:45.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.475+0000 7f1e9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:45.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.475+0000 7f1e9e7fc640 1 -- 172.21.15.18:0/522069225 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f1e94001c00 con 0x7f1ea00b1730 2024-03-25T18:06:45.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.475+0000 7f1ea5c40640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:45.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.475+0000 7f1ea5c40640 4 set_mon_vals no callback set 2024-03-25T18:06:45.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.475+0000 7f1e9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:45.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.475+0000 7f1e9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:06:45.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.475+0000 7f1e9e7fc640 10 monclient: dump: 2024-03-25T18:06:45.477 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:45.477 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:45.477 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:45.477 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:45.477 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:45.477 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:45.477 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:45.477 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:45.477 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:45.478 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:45.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.475+0000 7f1ea5c40640 10 monclient: shutdown 2024-03-25T18:06:45.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.475+0000 7f1ea5c40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:45.479 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.475+0000 7f1ea5c40640 1 -- 172.21.15.18:0/522069225 >> v1:172.21.15.18:6790/0 conn(0x7f1ea00b1730 legacy=0x7f1ea00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:45.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.476+0000 7f1ea5c40640 1 -- 172.21.15.18:0/522069225 shutdown_connections 2024-03-25T18:06:45.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.476+0000 7f1ea5c40640 1 -- 172.21.15.18:0/522069225 >> 172.21.15.18:0/522069225 conn(0x7f1ea00617a0 msgr2=0x7f1ea0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:45.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.476+0000 7f1ea5c40640 1 -- 172.21.15.18:0/522069225 shutdown_connections 2024-03-25T18:06:45.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.476+0000 7f1ea5c40640 1 -- 172.21.15.18:0/522069225 wait complete. 2024-03-25T18:06:45.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.476+0000 7f1ea5c40640 10 monclient: build_initial_monmap 2024-03-25T18:06:45.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.476+0000 7f1ea5c40640 10 monclient: monmap: 2024-03-25T18:06:45.480 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:06:45.480 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:45.480 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:06:45.480 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:06:45.480 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:45.480 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:06:45.480 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:06:45.480 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:06:45.481 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:06:45.481 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:45.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.477+0000 7f1ea5c40640 1 librados: starting msgr at 2024-03-25T18:06:45.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.477+0000 7f1ea5c40640 1 librados: starting objecter 2024-03-25T18:06:45.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.477+0000 7f1ea5c40640 1 Processor -- start 2024-03-25T18:06:45.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.477+0000 7f1ea5c40640 1 -- start start 2024-03-25T18:06:45.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.477+0000 7f1ea5c40640 1 librados: setting wanted keys 2024-03-25T18:06:45.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.477+0000 7f1ea5c40640 1 librados: calling monclient init 2024-03-25T18:06:45.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.477+0000 7f1ea5c40640 10 monclient: init 2024-03-25T18:06:45.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.478+0000 7f1ea5c40640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:45.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.478+0000 7f1ea5c40640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:06:45.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.478+0000 7f1ea5c40640 10 monclient(hunting): picked mon.noname-b con 0x7f1ea00b1730 addr v1:172.21.15.169:6789/0 2024-03-25T18:06:45.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.478+0000 7f1ea5c40640 10 monclient(hunting): picked mon.noname-c con 0x7f1ea00b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T18:06:45.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.478+0000 7f1ea5c40640 10 monclient(hunting): picked mon.noname-a con 0x7f1ea014be90 addr v1:172.21.15.18:6789/0 2024-03-25T18:06:45.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.478+0000 7f1ea5c40640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea0145600 con 0x7f1ea014be90 2024-03-25T18:06:45.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.478+0000 7f1ea5c40640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea0140520 con 0x7f1ea00b1730 2024-03-25T18:06:45.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.478+0000 7f1ea5c40640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea013d800 con 0x7f1ea00b4d60 2024-03-25T18:06:45.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.478+0000 7f1ea5c40640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:45.482 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.478+0000 7f1ea5c40640 10 monclient(hunting): authenticate will time out at 7534.057075s 2024-03-25T18:06:45.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.490+0000 7f1e9f7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f1ea00b1730 0x7f1ea013d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:42192/0 (socket says 172.21.15.18:42192) 2024-03-25T18:06:45.492 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.490+0000 7f1e9f7fe640 1 -- 172.21.15.18:0/1401400053 learned_addr learned my addr 172.21.15.18:0/1401400053 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:06:45.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.493+0000 7f1e87fff640 10 client.?.objecter ms_handle_connect 0x7f1ea014be90 2024-03-25T18:06:45.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.493+0000 7f1e87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:45.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.499+0000 7f1e87fff640 10 client.?.objecter ms_handle_connect 0x7f1ea00b1730 2024-03-25T18:06:45.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.499+0000 7f1e87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:45.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.500+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609714812 0 0) 0x7f1ea0145600 con 0x7f1ea014be90 2024-03-25T18:06:45.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.500+0000 7f1e87fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:45.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.500+0000 7f1e87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:45.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.500+0000 7f1e87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:45.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.500+0000 7f1e87fff640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:06:45.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.500+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e8c004eb0 con 0x7f1ea014be90 2024-03-25T18:06:45.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.505+0000 7f1e87fff640 10 client.?.objecter ms_handle_connect 0x7f1ea00b4d60 2024-03-25T18:06:45.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.505+0000 7f1e87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:45.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.514+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398574798 0 0) 0x7f1ea0140520 con 0x7f1ea00b1730 2024-03-25T18:06:45.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.514+0000 7f1e87fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:45.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.514+0000 7f1e87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:45.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.514+0000 7f1e87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:45.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.514+0000 7f1e87fff640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:06:45.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.514+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea0145600 con 0x7f1ea00b1730 2024-03-25T18:06:45.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.520+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1878165965 0 0) 0x7f1e8c004eb0 con 0x7f1ea014be90 2024-03-25T18:06:45.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.520+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ea0140520 con 0x7f1ea014be90 2024-03-25T18:06:45.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.520+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f1e98001b90 con 0x7f1ea014be90 2024-03-25T18:06:45.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.520+0000 7f1e87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:45.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.520+0000 7f1e87fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:06:45.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.520+0000 7f1e87fff640 10 monclient(hunting): dump: 2024-03-25T18:06:45.522 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:45.522 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:45.522 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:45.522 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:45.522 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:45.522 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:45.522 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:45.522 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:45.522 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:45.522 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:45.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.520+0000 7f1e87fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:45.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.523+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404946474 0 0) 0x7f1ea013d800 con 0x7f1ea00b4d60 2024-03-25T18:06:45.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.523+0000 7f1e87fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:45.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.523+0000 7f1e87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:45.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.523+0000 7f1e87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:45.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.523+0000 7f1e87fff640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:06:45.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.523+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e8c004eb0 con 0x7f1ea00b4d60 2024-03-25T18:06:45.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.537+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3916426346 0 0) 0x7f1ea0140520 con 0x7f1ea014be90 2024-03-25T18:06:45.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.537+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 >> v1:172.21.15.18:6790/0 conn(0x7f1ea00b4d60 legacy=0x7f1ea012fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:45.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.537+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 >> v1:172.21.15.169:6789/0 conn(0x7f1ea00b1730 legacy=0x7f1ea013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:45.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.537+0000 7f1e87fff640 10 monclient: _finish_hunting 0 2024-03-25T18:06:45.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.537+0000 7f1e87fff640 1 monclient: found mon.a 2024-03-25T18:06:45.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.537+0000 7f1e87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:45.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.537+0000 7f1e87fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:06:45.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.537+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea01448a0 con 0x7f1ea014be90 2024-03-25T18:06:45.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.537+0000 7f1e87fff640 10 monclient: _finish_auth 0 2024-03-25T18:06:45.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.537+0000 7f1e87fff640 10 monclient: _check_auth_tickets 2024-03-25T18:06:45.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.537+0000 7f1e87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:45.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.537+0000 7f1ea5c40640 5 monclient: authenticate success, global_id 4695 2024-03-25T18:06:45.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.538+0000 7f1ea5c40640 10 monclient: _renew_subs 2024-03-25T18:06:45.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.538+0000 7f1ea5c40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:06:45.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.538+0000 7f1ea5c40640 1 -- 172.21.15.18:0/1401400053 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea013eeb0 con 0x7f1ea014be90 2024-03-25T18:06:45.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.538+0000 7f1ea5c40640 10 client.4695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:45.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.538+0000 7f1ea5c40640 10 monclient: _renew_subs 2024-03-25T18:06:45.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.538+0000 7f1ea5c40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:06:45.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.538+0000 7f1ea5c40640 1 -- 172.21.15.18:0/1401400053 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea01444d0 con 0x7f1ea014be90 2024-03-25T18:06:45.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.538+0000 7f1ea5c40640 1 librados: init done 2024-03-25T18:06:45.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.539+0000 7f1ea5c40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:45.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.539+0000 7f1ea5c40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:45.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.539+0000 7f1ea5c40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:06:45.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.539+0000 7f1ea5c40640 1 -- 172.21.15.18:0/1401400053 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea01401f0 con 0x7f1ea014be90 2024-03-25T18:06:45.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.880+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e98002080 con 0x7f1ea014be90 2024-03-25T18:06:45.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.880+0000 7f1e87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:45.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.880+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f1e98003b30 con 0x7f1ea014be90 2024-03-25T18:06:45.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.880+0000 7f1e87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:45.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.880+0000 7f1e87fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:06:45.882 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.880+0000 7f1e87fff640 10 monclient: dump: 2024-03-25T18:06:45.882 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:45.882 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:45.882 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:45.882 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:45.882 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:45.883 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:45.883 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:45.883 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:45.883 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:45.883 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:45.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.880+0000 7f1e9dffb640 4 set_mon_vals no callback set 2024-03-25T18:06:45.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.881+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f1e98003d50 con 0x7f1ea014be90 2024-03-25T18:06:45.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.881+0000 7f1e9f7fe640 10 monclient: get_auth_request con 0x7f1e8c06ddb0 auth_method 0 2024-03-25T18:06:45.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.881+0000 7f1e87fff640 10 client.4695.objecter ms_handle_connect 0x7f1e8c06ddb0 2024-03-25T18:06:45.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.888+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 865634137 0 0) 0x7f1e980537d0 con 0x7f1ea014be90 2024-03-25T18:06:45.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.888+0000 7f1e87fff640 10 client.4695.objecter ms_dispatch 0x7f1ea0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:06:45.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.888+0000 7f1e87fff640 3 client.4695.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:06:45.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.888+0000 7f1e87fff640 3 client.4695.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:06:45.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.888+0000 7f1e87fff640 20 client.4695.objecter dump_active .. 0 homeless 2024-03-25T18:06:45.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.888+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1e9804f2e0 con 0x7f1ea014be90 2024-03-25T18:06:45.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.888+0000 7f1e87fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:45.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:45.888+0000 7f1e87fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:46.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.142+0000 7f1ea5c40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:46.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.142+0000 7f1ea5c40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:46.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.142+0000 7f1ea5c40640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:06:46.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.142+0000 7f1ea5c40640 1 -- 172.21.15.18:0/1401400053 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ea00b5cf0 con 0x7f1ea014be90 2024-03-25T18:06:46.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.228+0000 7f1e87fff640 1 -- 172.21.15.18:0/1401400053 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 678331228) 0x7f1e98051210 con 0x7f1ea014be90 2024-03-25T18:06:46.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.228+0000 7f1e87fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:46.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.228+0000 7f1e87fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:46.230 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:06:46.230 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T18:06:29.006918+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:06:29.006918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:46.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.231+0000 7f1ea5c40640 10 librados: watch_flush enter 2024-03-25T18:06:46.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.231+0000 7f1ea5c40640 10 librados: watch_flush exit 2024-03-25T18:06:46.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.231+0000 7f1ea5c40640 20 client.4695.objecter shutdown clearing up homeless session... 2024-03-25T18:06:46.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.231+0000 7f1ea5c40640 10 client.4695.objecter successfully canceled tick 2024-03-25T18:06:46.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.231+0000 7f1ea5c40640 1 -- 172.21.15.18:0/1401400053 >> v1:172.21.15.18:6800/3721244451 conn(0x7f1e8c06ddb0 legacy=0x7f1e8c070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:46.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.231+0000 7f1ea5c40640 10 monclient: shutdown 2024-03-25T18:06:46.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.231+0000 7f1ea5c40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:46.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.231+0000 7f1ea5c40640 1 -- 172.21.15.18:0/1401400053 >> v1:172.21.15.18:6789/0 conn(0x7f1ea014be90 legacy=0x7f1ea01e3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:46.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.231+0000 7f1ea5c40640 1 -- 172.21.15.18:0/1401400053 shutdown_connections 2024-03-25T18:06:46.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.231+0000 7f1ea5c40640 1 -- 172.21.15.18:0/1401400053 >> 172.21.15.18:0/1401400053 conn(0x7f1ea00617a0 msgr2=0x7f1ea00b64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:46.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.231+0000 7f1ea5c40640 1 -- 172.21.15.18:0/1401400053 shutdown_connections 2024-03-25T18:06:46.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.231+0000 7f1ea5c40640 1 -- 172.21.15.18:0/1401400053 wait complete. 2024-03-25T18:06:46.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:46.231+0000 7f1ea5c40640 1 librados: shutdown 2024-03-25T18:06:46.244 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:49.137 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:46.929 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:49.183 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:46.931 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:49.186 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:46.935 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:49.203 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:46.995 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:49.228 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:10:46.977 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:49.241 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:47.034 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:49.244 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:10:47.036 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:49.254 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:47.047 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:49.283 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:10:47.076 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:53.051 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:53.150 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.148+0000 7f5c71281640 10 monclient: get_monmap_and_config 2024-03-25T18:06:53.150 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.148+0000 7f5c71281640 10 monclient: build_initial_monmap 2024-03-25T18:06:53.150 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.149+0000 7f5c71281640 10 monclient: monmap: 2024-03-25T18:06:53.150 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:06:53.151 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:53.151 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:06:53.150357+0000 2024-03-25T18:06:53.151 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:06:53.150357+0000 2024-03-25T18:06:53.151 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:53.151 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:53.151 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:06:53.151 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:06:53.151 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:06:53.151 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:53.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.149+0000 7f5c71281640 1 Processor -- start 2024-03-25T18:06:53.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.149+0000 7f5c71281640 1 -- start start 2024-03-25T18:06:53.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.149+0000 7f5c71281640 10 monclient: init 2024-03-25T18:06:53.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.150+0000 7f5c71281640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:53.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.150+0000 7f5c71281640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:06:53.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.150+0000 7f5c71281640 10 monclient(hunting): picked mon.noname-c con 0x7f5c6c0a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:06:53.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.150+0000 7f5c71281640 10 monclient(hunting): picked mon.noname-b con 0x7f5c6c13f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:06:53.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.150+0000 7f5c71281640 10 monclient(hunting): picked mon.noname-a con 0x7f5c6c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:06:53.153 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.150+0000 7f5c71281640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c6c142520 con 0x7f5c6c1430c0 2024-03-25T18:06:53.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.150+0000 7f5c71281640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c6c0b28a0 con 0x7f5c6c0a98f0 2024-03-25T18:06:53.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.150+0000 7f5c71281640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c6c0b2a30 con 0x7f5c6c13f710 2024-03-25T18:06:53.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.150+0000 7f5c71281640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:53.154 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.150+0000 7f5c71281640 10 monclient(hunting): authenticate will time out at 7541.728923s 2024-03-25T18:06:53.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.161+0000 7f5c6b7fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f5c6c1430c0 0x7f5c6c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:36872/0 (socket says 172.21.15.18:36872) 2024-03-25T18:06:53.162 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.161+0000 7f5c6b7fe640 1 -- 172.21.15.18:0/1791776339 learned_addr learned my addr 172.21.15.18:0/1791776339 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:06:53.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.181+0000 7f5c69ffb640 1 -- 172.21.15.18:0/1791776339 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189496077 0 0) 0x7f5c6c142520 con 0x7f5c6c1430c0 2024-03-25T18:06:53.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.181+0000 7f5c69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:06:53.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.181+0000 7f5c69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:53.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.181+0000 7f5c69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:53.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.181+0000 7f5c69ffb640 10 monclient(hunting): my global_id is 4698 2024-03-25T18:06:53.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.181+0000 7f5c69ffb640 1 -- 172.21.15.18:0/1791776339 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c54003520 con 0x7f5c6c1430c0 2024-03-25T18:06:53.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.188+0000 7f5c69ffb640 1 -- 172.21.15.18:0/1791776339 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 90522360 0 0) 0x7f5c54003520 con 0x7f5c6c1430c0 2024-03-25T18:06:53.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.188+0000 7f5c69ffb640 1 -- 172.21.15.18:0/1791776339 >> v1:172.21.15.18:6790/0 conn(0x7f5c6c13f710 legacy=0x7f5c6c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:06:53.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.188+0000 7f5c69ffb640 1 -- 172.21.15.18:0/1791776339 >> v1:172.21.15.169:6789/0 conn(0x7f5c6c0a98f0 legacy=0x7f5c6c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:06:53.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.188+0000 7f5c69ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:06:53.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.188+0000 7f5c69ffb640 1 monclient: found mon.noname-a 2024-03-25T18:06:53.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.188+0000 7f5c69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:53.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.188+0000 7f5c69ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:06:53.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.188+0000 7f5c69ffb640 1 -- 172.21.15.18:0/1791776339 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6c0b44c0 con 0x7f5c6c1430c0 2024-03-25T18:06:53.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.188+0000 7f5c69ffb640 10 monclient: _finish_auth 0 2024-03-25T18:06:53.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.188+0000 7f5c69ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:06:53.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.188+0000 7f5c69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:53.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.188+0000 7f5c69ffb640 1 -- 172.21.15.18:0/1791776339 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5c64001a10 con 0x7f5c6c1430c0 2024-03-25T18:06:53.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.188+0000 7f5c69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:53.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.188+0000 7f5c69ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:06:53.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.188+0000 7f5c69ffb640 10 monclient: dump: 2024-03-25T18:06:53.191 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:53.191 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:53.192 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:53.192 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:53.192 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:53.192 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:53.192 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:53.192 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:53.192 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:53.192 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:53.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.188+0000 7f5c71281640 5 monclient: authenticate success, global_id 4698 2024-03-25T18:06:53.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.188+0000 7f5c71281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:53.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.200+0000 7f5c69ffb640 1 -- 172.21.15.18:0/1791776339 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c64001be0 con 0x7f5c6c1430c0 2024-03-25T18:06:53.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.200+0000 7f5c69ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:53.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.200+0000 7f5c69ffb640 1 -- 172.21.15.18:0/1791776339 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5c64002030 con 0x7f5c6c1430c0 2024-03-25T18:06:53.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.200+0000 7f5c69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:53.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.200+0000 7f5c69ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:06:53.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.200+0000 7f5c69ffb640 10 monclient: dump: 2024-03-25T18:06:53.202 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:53.202 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:53.202 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:53.203 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:53.203 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:53.203 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:53.203 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:53.203 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:53.203 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:53.203 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:53.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.200+0000 7f5c71281640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:53.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.200+0000 7f5c71281640 4 set_mon_vals no callback set 2024-03-25T18:06:53.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.201+0000 7f5c71281640 10 monclient: shutdown 2024-03-25T18:06:53.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.201+0000 7f5c71281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:53.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.201+0000 7f5c71281640 1 -- 172.21.15.18:0/1791776339 >> v1:172.21.15.18:6789/0 conn(0x7f5c6c1430c0 legacy=0x7f5c6c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:53.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.201+0000 7f5c71281640 1 -- 172.21.15.18:0/1791776339 shutdown_connections 2024-03-25T18:06:53.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.201+0000 7f5c71281640 1 -- 172.21.15.18:0/1791776339 >> 172.21.15.18:0/1791776339 conn(0x7f5c6c0b7ea0 msgr2=0x7f5c6c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:53.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.201+0000 7f5c71281640 1 -- 172.21.15.18:0/1791776339 shutdown_connections 2024-03-25T18:06:53.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.201+0000 7f5c71281640 1 -- 172.21.15.18:0/1791776339 wait complete. 2024-03-25T18:06:53.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.202+0000 7f5c71281640 10 monclient: build_initial_monmap 2024-03-25T18:06:53.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.202+0000 7f5c71281640 10 monclient: monmap: 2024-03-25T18:06:53.207 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:06:53.207 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:53.207 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:06:53.207 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:06:53.207 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:53.207 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:06:53.207 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:06:53.207 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:06:53.207 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:06:53.207 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:53.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.202+0000 7f5c71281640 1 librados: starting msgr at 2024-03-25T18:06:53.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.202+0000 7f5c71281640 1 librados: starting objecter 2024-03-25T18:06:53.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.202+0000 7f5c71281640 1 Processor -- start 2024-03-25T18:06:53.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.202+0000 7f5c71281640 1 -- start start 2024-03-25T18:06:53.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.202+0000 7f5c71281640 1 librados: setting wanted keys 2024-03-25T18:06:53.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.202+0000 7f5c71281640 1 librados: calling monclient init 2024-03-25T18:06:53.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.202+0000 7f5c71281640 10 monclient: init 2024-03-25T18:06:53.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.203+0000 7f5c71281640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:53.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.203+0000 7f5c71281640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:06:53.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.203+0000 7f5c71281640 10 monclient(hunting): picked mon.noname-b con 0x7f5c6c0a98f0 addr v1:172.21.15.169:6789/0 2024-03-25T18:06:53.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.203+0000 7f5c71281640 10 monclient(hunting): picked mon.noname-c con 0x7f5c6c13f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:06:53.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.203+0000 7f5c71281640 10 monclient(hunting): picked mon.noname-a con 0x7f5c6c1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:06:53.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.203+0000 7f5c71281640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c6c0abcf0 con 0x7f5c6c1430c0 2024-03-25T18:06:53.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.203+0000 7f5c71281640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c6c138360 con 0x7f5c6c0a98f0 2024-03-25T18:06:53.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.203+0000 7f5c71281640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c6c1d8320 con 0x7f5c6c13f710 2024-03-25T18:06:53.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.203+0000 7f5c71281640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:53.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.203+0000 7f5c71281640 10 monclient(hunting): authenticate will time out at 7541.782283s 2024-03-25T18:06:53.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.215+0000 7f5c6affd640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f5c6c0a98f0 0x7f5c6c1389a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:46574/0 (socket says 172.21.15.18:46574) 2024-03-25T18:06:53.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.215+0000 7f5c6affd640 1 -- 172.21.15.18:0/758185369 learned_addr learned my addr 172.21.15.18:0/758185369 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:06:53.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.222+0000 7f5c537fe640 10 client.?.objecter ms_handle_connect 0x7f5c6c0a98f0 2024-03-25T18:06:53.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.222+0000 7f5c537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:53.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.226+0000 7f5c537fe640 10 client.?.objecter ms_handle_connect 0x7f5c6c1430c0 2024-03-25T18:06:53.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.226+0000 7f5c537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:53.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.234+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945812688 0 0) 0x7f5c6c0abcf0 con 0x7f5c6c1430c0 2024-03-25T18:06:53.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.234+0000 7f5c537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:53.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.234+0000 7f5c537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:53.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.234+0000 7f5c537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:53.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.234+0000 7f5c537fe640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:06:53.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.234+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c58004eb0 con 0x7f5c6c1430c0 2024-03-25T18:06:53.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.235+0000 7f5c537fe640 10 client.?.objecter ms_handle_connect 0x7f5c6c13f710 2024-03-25T18:06:53.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.235+0000 7f5c537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:53.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.239+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212230394 0 0) 0x7f5c6c138360 con 0x7f5c6c0a98f0 2024-03-25T18:06:53.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.239+0000 7f5c537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:53.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.239+0000 7f5c537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:53.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.239+0000 7f5c537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:53.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.239+0000 7f5c537fe640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:06:53.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.239+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c6c0abcf0 con 0x7f5c6c0a98f0 2024-03-25T18:06:53.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.243+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582851641 0 0) 0x7f5c6c1d8320 con 0x7f5c6c13f710 2024-03-25T18:06:53.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.243+0000 7f5c537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:53.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.243+0000 7f5c537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:53.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.243+0000 7f5c537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:53.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.243+0000 7f5c537fe640 10 monclient(hunting): my global_id is 4706 2024-03-25T18:06:53.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.243+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c6c138360 con 0x7f5c6c13f710 2024-03-25T18:06:53.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.247+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1153487458 0 0) 0x7f5c58004eb0 con 0x7f5c6c1430c0 2024-03-25T18:06:53.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.247+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c6c1d8320 con 0x7f5c6c1430c0 2024-03-25T18:06:53.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.247+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5c64001ef0 con 0x7f5c6c1430c0 2024-03-25T18:06:53.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.247+0000 7f5c537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:53.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.247+0000 7f5c537fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:06:53.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.247+0000 7f5c537fe640 10 monclient(hunting): dump: 2024-03-25T18:06:53.249 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:53.249 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:53.249 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:53.249 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:53.249 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:53.249 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:53.250 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:53.250 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:53.250 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:53.250 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:53.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.247+0000 7f5c537fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:53.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.252+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4292915955 0 0) 0x7f5c6c138360 con 0x7f5c6c13f710 2024-03-25T18:06:53.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.252+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c58004eb0 con 0x7f5c6c13f710 2024-03-25T18:06:53.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.252+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5c600017d0 con 0x7f5c6c13f710 2024-03-25T18:06:53.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.252+0000 7f5c537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:53.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.252+0000 7f5c537fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:06:53.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.252+0000 7f5c537fe640 10 monclient(hunting): dump: 2024-03-25T18:06:53.254 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:53.255 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:53.255 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:53.255 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:53.255 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:53.255 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:53.255 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:53.255 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:53.255 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:53.255 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:53.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2649368941 0 0) 0x7f5c6c0abcf0 con 0x7f5c6c0a98f0 2024-03-25T18:06:53.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c6c138360 con 0x7f5c6c0a98f0 2024-03-25T18:06:53.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5c5c0018e0 con 0x7f5c6c0a98f0 2024-03-25T18:06:53.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:53.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c537fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:53.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c537fe640 10 monclient(hunting): dump: 2024-03-25T18:06:53.263 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:53.263 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:53.263 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:53.263 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:53.263 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:53.263 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:53.263 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:53.264 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:53.264 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:53.264 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:53.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 488774431 0 0) 0x7f5c6c1d8320 con 0x7f5c6c1430c0 2024-03-25T18:06:53.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 >> v1:172.21.15.18:6790/0 conn(0x7f5c6c13f710 legacy=0x7f5c6c1d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:53.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 >> v1:172.21.15.169:6789/0 conn(0x7f5c6c0a98f0 legacy=0x7f5c6c1389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:53.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c537fe640 10 monclient: _finish_hunting 0 2024-03-25T18:06:53.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c537fe640 1 monclient: found mon.a 2024-03-25T18:06:53.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:53.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c537fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:06:53.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6c1347b0 con 0x7f5c6c1430c0 2024-03-25T18:06:53.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c537fe640 10 monclient: _finish_auth 0 2024-03-25T18:06:53.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c537fe640 10 monclient: _check_auth_tickets 2024-03-25T18:06:53.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:53.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c71281640 5 monclient: authenticate success, global_id 4701 2024-03-25T18:06:53.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c71281640 10 monclient: _renew_subs 2024-03-25T18:06:53.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c71281640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:06:53.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c71281640 1 -- 172.21.15.18:0/758185369 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c6c134a60 con 0x7f5c6c1430c0 2024-03-25T18:06:53.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c71281640 10 client.4701.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:53.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c71281640 10 monclient: _renew_subs 2024-03-25T18:06:53.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c71281640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:06:53.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c71281640 1 -- 172.21.15.18:0/758185369 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c6c137e70 con 0x7f5c6c1430c0 2024-03-25T18:06:53.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.260+0000 7f5c71281640 1 librados: init done 2024-03-25T18:06:53.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.262+0000 7f5c71281640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:53.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.262+0000 7f5c71281640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:53.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.262+0000 7f5c71281640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:06:53.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.262+0000 7f5c71281640 1 -- 172.21.15.18:0/758185369 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c6c0627b0 con 0x7f5c6c1430c0 2024-03-25T18:06:53.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.589+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c64004e10 con 0x7f5c6c1430c0 2024-03-25T18:06:53.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.589+0000 7f5c537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:53.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.589+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f5c640015d0 con 0x7f5c6c1430c0 2024-03-25T18:06:53.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.589+0000 7f5c537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:53.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.589+0000 7f5c537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:06:53.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.589+0000 7f5c537fe640 10 monclient: dump: 2024-03-25T18:06:53.592 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:06:53.592 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:06:53.592 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:53.592 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:06:53.592 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:53.592 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:06:53.592 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:06:53.592 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:06:53.592 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:06:53.592 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:06:53.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.589+0000 7f5c697fa640 4 set_mon_vals no callback set 2024-03-25T18:06:53.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.590+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f5c6401af90 con 0x7f5c6c1430c0 2024-03-25T18:06:53.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.591+0000 7f5c6affd640 10 monclient: get_auth_request con 0x7f5c5806e060 auth_method 0 2024-03-25T18:06:53.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.591+0000 7f5c537fe640 10 client.4701.objecter ms_handle_connect 0x7f5c5806e060 2024-03-25T18:06:53.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.600+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 865634137 0 0) 0x7f5c64054b70 con 0x7f5c6c1430c0 2024-03-25T18:06:53.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.600+0000 7f5c537fe640 10 client.4701.objecter ms_dispatch 0x7f5c6c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:06:53.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.600+0000 7f5c537fe640 3 client.4701.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:06:53.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.600+0000 7f5c537fe640 3 client.4701.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:06:53.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.600+0000 7f5c537fe640 20 client.4701.objecter dump_active .. 0 homeless 2024-03-25T18:06:53.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.600+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5c64050680 con 0x7f5c6c1430c0 2024-03-25T18:06:53.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.600+0000 7f5c537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:53.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.600+0000 7f5c537fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:53.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.855+0000 7f5c71281640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:53.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.855+0000 7f5c71281640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:53.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.855+0000 7f5c71281640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:06:53.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.855+0000 7f5c71281640 1 -- 172.21.15.18:0/758185369 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c6c0b1740 con 0x7f5c6c1430c0 2024-03-25T18:06:53.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.893+0000 7f5c537fe640 1 -- 172.21.15.18:0/758185369 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 678331228) 0x7f5c640525b0 con 0x7f5c6c1430c0 2024-03-25T18:06:53.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.893+0000 7f5c537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:53.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.893+0000 7f5c537fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:53.895 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:06:53.895 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T18:06:29.006918+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:06:29.006918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:53.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.895+0000 7f5c71281640 10 librados: watch_flush enter 2024-03-25T18:06:53.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.895+0000 7f5c71281640 10 librados: watch_flush exit 2024-03-25T18:06:53.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.895+0000 7f5c71281640 20 client.4701.objecter shutdown clearing up homeless session... 2024-03-25T18:06:53.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.895+0000 7f5c71281640 10 client.4701.objecter successfully canceled tick 2024-03-25T18:06:53.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.895+0000 7f5c71281640 1 -- 172.21.15.18:0/758185369 >> v1:172.21.15.18:6800/3721244451 conn(0x7f5c5806e060 legacy=0x7f5c58070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:53.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.895+0000 7f5c71281640 10 monclient: shutdown 2024-03-25T18:06:53.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.895+0000 7f5c71281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:53.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.895+0000 7f5c71281640 1 -- 172.21.15.18:0/758185369 >> v1:172.21.15.18:6789/0 conn(0x7f5c6c1430c0 legacy=0x7f5c6c1340a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:53.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.895+0000 7f5c71281640 1 -- 172.21.15.18:0/758185369 shutdown_connections 2024-03-25T18:06:53.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.896+0000 7f5c71281640 1 -- 172.21.15.18:0/758185369 >> 172.21.15.18:0/758185369 conn(0x7f5c6c0b7ea0 msgr2=0x7f5c6c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:53.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.896+0000 7f5c71281640 1 -- 172.21.15.18:0/758185369 shutdown_connections 2024-03-25T18:06:53.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.896+0000 7f5c71281640 1 -- 172.21.15.18:0/758185369 wait complete. 2024-03-25T18:06:53.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:06:53.896+0000 7f5c71281640 1 librados: shutdown 2024-03-25T18:06:53.909 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:00.716 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:00.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.830+0000 7fc2047ba640 10 monclient: get_monmap_and_config 2024-03-25T18:07:00.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.830+0000 7fc2047ba640 10 monclient: build_initial_monmap 2024-03-25T18:07:00.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.830+0000 7fc2047ba640 10 monclient: monmap: 2024-03-25T18:07:00.832 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:07:00.832 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:00.832 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:07:00.832033+0000 2024-03-25T18:07:00.832 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:07:00.832033+0000 2024-03-25T18:07:00.832 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:00.832 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:00.832 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:07:00.833 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:07:00.833 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:07:00.833 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:00.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.831+0000 7fc2047ba640 1 Processor -- start 2024-03-25T18:07:00.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.831+0000 7fc2047ba640 1 -- start start 2024-03-25T18:07:00.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.831+0000 7fc2047ba640 10 monclient: init 2024-03-25T18:07:00.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.832+0000 7fc2047ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:00.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.832+0000 7fc2047ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:07:00.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.832+0000 7fc2047ba640 10 monclient(hunting): picked mon.noname-c con 0x7fc1fc0b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T18:07:00.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.832+0000 7fc2047ba640 10 monclient(hunting): picked mon.noname-b con 0x7fc1fc0b1730 addr v1:172.21.15.18:6790/0 2024-03-25T18:07:00.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.832+0000 7fc2047ba640 10 monclient(hunting): picked mon.noname-a con 0x7fc1fc14be90 addr v1:172.21.15.18:6789/0 2024-03-25T18:07:00.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.832+0000 7fc2047ba640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1fc0af940 con 0x7fc1fc14be90 2024-03-25T18:07:00.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.832+0000 7fc2047ba640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1fc0b0a70 con 0x7fc1fc0b4d60 2024-03-25T18:07:00.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.832+0000 7fc2047ba640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1fc0b0d50 con 0x7fc1fc0b1730 2024-03-25T18:07:00.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.832+0000 7fc2047ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:00.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.832+0000 7fc2047ba640 10 monclient(hunting): authenticate will time out at 7549.410922s 2024-03-25T18:07:00.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.836+0000 7fc201d55640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fc1fc0b1730 0x7fc1fc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:56632/0 (socket says 172.21.15.18:56632) 2024-03-25T18:07:00.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.836+0000 7fc201d55640 1 -- 172.21.15.18:0/1946065288 learned_addr learned my addr 172.21.15.18:0/1946065288 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:07:00.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.848+0000 7fc201554640 1 -- 172.21.15.18:0/1946065288 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746216871 0 0) 0x7fc1fc0b0d50 con 0x7fc1fc0b1730 2024-03-25T18:07:00.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.848+0000 7fc201554640 10 monclient(hunting): _negotiate 2024-03-25T18:07:00.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.848+0000 7fc201554640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:00.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.848+0000 7fc201554640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:00.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.848+0000 7fc201554640 10 monclient(hunting): my global_id is 4709 2024-03-25T18:07:00.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.848+0000 7fc201554640 1 -- 172.21.15.18:0/1946065288 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e8003520 con 0x7fc1fc0b1730 2024-03-25T18:07:00.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.860+0000 7fc201554640 1 -- 172.21.15.18:0/1946065288 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 496221524 0 0) 0x7fc1e8003520 con 0x7fc1fc0b1730 2024-03-25T18:07:00.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.860+0000 7fc201554640 1 -- 172.21.15.18:0/1946065288 >> v1:172.21.15.169:6789/0 conn(0x7fc1fc0b4d60 legacy=0x7fc1fc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:00.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.860+0000 7fc201554640 1 -- 172.21.15.18:0/1946065288 >> v1:172.21.15.18:6789/0 conn(0x7fc1fc14be90 legacy=0x7fc1fc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:00.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.860+0000 7fc201554640 10 monclient: _finish_hunting 0 2024-03-25T18:07:00.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.860+0000 7fc201554640 1 monclient: found mon.noname-b 2024-03-25T18:07:00.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.860+0000 7fc201554640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:00.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.861+0000 7fc201554640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:07:00.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.861+0000 7fc201554640 1 -- 172.21.15.18:0/1946065288 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1fc0b1030 con 0x7fc1fc0b1730 2024-03-25T18:07:00.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.861+0000 7fc201554640 10 monclient: _finish_auth 0 2024-03-25T18:07:00.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.861+0000 7fc201554640 10 monclient: _check_auth_tickets 2024-03-25T18:07:00.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.861+0000 7fc201554640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:00.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.861+0000 7fc201554640 1 -- 172.21.15.18:0/1946065288 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc1ec001690 con 0x7fc1fc0b1730 2024-03-25T18:07:00.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.861+0000 7fc2047ba640 5 monclient: authenticate success, global_id 4709 2024-03-25T18:07:00.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.861+0000 7fc2047ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:00.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.861+0000 7fc201554640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:00.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.861+0000 7fc201554640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:00.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.861+0000 7fc201554640 10 monclient: dump: 2024-03-25T18:07:00.863 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:00.863 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:00.863 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:00.864 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:00.864 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:00.864 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:00.864 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:00.864 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:00.864 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:00.864 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:00.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.861+0000 7fc2047ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:00.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.869+0000 7fc201554640 1 -- 172.21.15.18:0/1946065288 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1ec0017f0 con 0x7fc1fc0b1730 2024-03-25T18:07:00.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.869+0000 7fc201554640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:00.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.869+0000 7fc201554640 1 -- 172.21.15.18:0/1946065288 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc1ec001c00 con 0x7fc1fc0b1730 2024-03-25T18:07:00.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.869+0000 7fc2047ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:00.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.869+0000 7fc2047ba640 4 set_mon_vals no callback set 2024-03-25T18:07:00.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.869+0000 7fc201554640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:00.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.869+0000 7fc201554640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:00.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.869+0000 7fc201554640 10 monclient: dump: 2024-03-25T18:07:00.871 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:00.871 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:00.871 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:00.871 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:00.871 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:00.871 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:00.872 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:00.872 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:00.872 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:00.872 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:00.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.870+0000 7fc2047ba640 10 monclient: shutdown 2024-03-25T18:07:00.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.870+0000 7fc2047ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:00.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.870+0000 7fc2047ba640 1 -- 172.21.15.18:0/1946065288 >> v1:172.21.15.18:6790/0 conn(0x7fc1fc0b1730 legacy=0x7fc1fc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:00.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.870+0000 7fc2047ba640 1 -- 172.21.15.18:0/1946065288 shutdown_connections 2024-03-25T18:07:00.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.870+0000 7fc2047ba640 1 -- 172.21.15.18:0/1946065288 >> 172.21.15.18:0/1946065288 conn(0x7fc1fc0617a0 msgr2=0x7fc1fc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:00.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.870+0000 7fc2047ba640 1 -- 172.21.15.18:0/1946065288 shutdown_connections 2024-03-25T18:07:00.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.870+0000 7fc2047ba640 1 -- 172.21.15.18:0/1946065288 wait complete. 2024-03-25T18:07:00.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.871+0000 7fc2047ba640 10 monclient: build_initial_monmap 2024-03-25T18:07:00.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.871+0000 7fc2047ba640 10 monclient: monmap: 2024-03-25T18:07:00.874 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:07:00.874 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:00.874 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:07:00.874 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:07:00.875 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:00.875 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:07:00.875 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:07:00.875 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:07:00.875 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:07:00.875 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:00.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.872+0000 7fc2047ba640 1 librados: starting msgr at 2024-03-25T18:07:00.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.872+0000 7fc2047ba640 1 librados: starting objecter 2024-03-25T18:07:00.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.872+0000 7fc2047ba640 1 Processor -- start 2024-03-25T18:07:00.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.872+0000 7fc2047ba640 1 -- start start 2024-03-25T18:07:00.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.872+0000 7fc2047ba640 1 librados: setting wanted keys 2024-03-25T18:07:00.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.872+0000 7fc2047ba640 1 librados: calling monclient init 2024-03-25T18:07:00.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.872+0000 7fc2047ba640 10 monclient: init 2024-03-25T18:07:00.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.872+0000 7fc2047ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:00.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.872+0000 7fc2047ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:07:00.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.872+0000 7fc2047ba640 10 monclient(hunting): picked mon.noname-b con 0x7fc1fc0b1730 addr v1:172.21.15.169:6789/0 2024-03-25T18:07:00.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.872+0000 7fc2047ba640 10 monclient(hunting): picked mon.noname-c con 0x7fc1fc0b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T18:07:00.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.872+0000 7fc2047ba640 10 monclient(hunting): picked mon.noname-a con 0x7fc1fc14be90 addr v1:172.21.15.18:6789/0 2024-03-25T18:07:00.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.873+0000 7fc2047ba640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1fc1dafb0 con 0x7fc1fc14be90 2024-03-25T18:07:00.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.873+0000 7fc2047ba640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1fc1432d0 con 0x7fc1fc0b1730 2024-03-25T18:07:00.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.873+0000 7fc2047ba640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1fc1dbbe0 con 0x7fc1fc0b4d60 2024-03-25T18:07:00.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.873+0000 7fc2047ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:00.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.873+0000 7fc2047ba640 10 monclient(hunting): authenticate will time out at 7549.451678s 2024-03-25T18:07:00.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.882+0000 7fc202556640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc1fc0b1730 0x7fc1fc1db4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:57636/0 (socket says 172.21.15.18:57636) 2024-03-25T18:07:00.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.882+0000 7fc202556640 1 -- 172.21.15.18:0/898030849 learned_addr learned my addr 172.21.15.18:0/898030849 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:07:00.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.882+0000 7fc1f2ffd640 10 client.?.objecter ms_handle_connect 0x7fc1fc0b1730 2024-03-25T18:07:00.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.882+0000 7fc1f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:00.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.890+0000 7fc1f2ffd640 10 client.?.objecter ms_handle_connect 0x7fc1fc14be90 2024-03-25T18:07:00.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.890+0000 7fc1f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:00.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.897+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950164308 0 0) 0x7fc1fc1432d0 con 0x7fc1fc0b1730 2024-03-25T18:07:00.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.897+0000 7fc1f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:00.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.897+0000 7fc1f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:00.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.897+0000 7fc1f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:00.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.897+0000 7fc1f2ffd640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:07:00.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.897+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e0004eb0 con 0x7fc1fc0b1730 2024-03-25T18:07:00.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.900+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126195011 0 0) 0x7fc1fc1dafb0 con 0x7fc1fc14be90 2024-03-25T18:07:00.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.900+0000 7fc1f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:00.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.900+0000 7fc1f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:00.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.900+0000 7fc1f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:00.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.900+0000 7fc1f2ffd640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:07:00.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.900+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1fc1432d0 con 0x7fc1fc14be90 2024-03-25T18:07:00.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.901+0000 7fc1f2ffd640 10 client.?.objecter ms_handle_connect 0x7fc1fc0b4d60 2024-03-25T18:07:00.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.901+0000 7fc1f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:00.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.911+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3525165977 0 0) 0x7fc1fc1432d0 con 0x7fc1fc14be90 2024-03-25T18:07:00.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.911+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1fc1dafb0 con 0x7fc1fc14be90 2024-03-25T18:07:00.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.911+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc1f4001810 con 0x7fc1fc14be90 2024-03-25T18:07:00.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.911+0000 7fc1f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:00.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.911+0000 7fc1f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:00.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.911+0000 7fc1f2ffd640 10 monclient(hunting): dump: 2024-03-25T18:07:00.913 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:00.913 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:00.913 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:00.913 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:00.913 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:00.913 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:00.914 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:00.914 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:00.914 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:00.914 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:00.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.911+0000 7fc1f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:00.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.912+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397025345 0 0) 0x7fc1fc1dbbe0 con 0x7fc1fc0b4d60 2024-03-25T18:07:00.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.912+0000 7fc1f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:00.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.912+0000 7fc1f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:00.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.912+0000 7fc1f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:00.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.912+0000 7fc1f2ffd640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:07:00.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.912+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1fc1432d0 con 0x7fc1fc0b4d60 2024-03-25T18:07:00.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.913+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1503936262 0 0) 0x7fc1e0004eb0 con 0x7fc1fc0b1730 2024-03-25T18:07:00.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.913+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1fc1dbbe0 con 0x7fc1fc0b1730 2024-03-25T18:07:00.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.913+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc1f8001db0 con 0x7fc1fc0b1730 2024-03-25T18:07:00.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.913+0000 7fc1f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:00.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.913+0000 7fc1f2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:00.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.913+0000 7fc1f2ffd640 10 monclient(hunting): dump: 2024-03-25T18:07:00.916 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:00.916 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:00.916 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:00.916 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:00.916 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:00.916 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:00.916 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:00.916 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:00.916 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:00.916 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:00.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.926+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2122262104 0 0) 0x7fc1fc1432d0 con 0x7fc1fc0b4d60 2024-03-25T18:07:00.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.926+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1e0004eb0 con 0x7fc1fc0b4d60 2024-03-25T18:07:00.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.926+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc1ec001b00 con 0x7fc1fc0b4d60 2024-03-25T18:07:00.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.926+0000 7fc1f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:00.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.926+0000 7fc1f2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:00.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.926+0000 7fc1f2ffd640 10 monclient(hunting): dump: 2024-03-25T18:07:00.927 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:00.927 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:00.927 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:00.927 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:00.928 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:00.928 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:00.928 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:00.928 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:00.928 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:00.928 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:00.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2867296759 0 0) 0x7fc1fc1dbbe0 con 0x7fc1fc0b1730 2024-03-25T18:07:00.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 >> v1:172.21.15.18:6790/0 conn(0x7fc1fc0b4d60 legacy=0x7fc1fc138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:00.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 >> v1:172.21.15.18:6789/0 conn(0x7fc1fc14be90 legacy=0x7fc1fc13b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:00.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc1f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:07:00.929 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc1f2ffd640 1 monclient: found mon.b 2024-03-25T18:07:00.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc1f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:00.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc1f2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:07:00.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1fc139670 con 0x7fc1fc0b1730 2024-03-25T18:07:00.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc1f2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:07:00.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc1f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:07:00.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc1f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:00.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc2047ba640 5 monclient: authenticate success, global_id 4693 2024-03-25T18:07:00.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc2047ba640 10 monclient: _renew_subs 2024-03-25T18:07:00.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc2047ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:07:00.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc2047ba640 1 -- 172.21.15.18:0/898030849 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1fc13bbe0 con 0x7fc1fc0b1730 2024-03-25T18:07:00.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc2047ba640 10 client.4693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:00.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc2047ba640 10 monclient: _renew_subs 2024-03-25T18:07:00.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc2047ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:07:00.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc2047ba640 1 -- 172.21.15.18:0/898030849 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1fc142a30 con 0x7fc1fc0b1730 2024-03-25T18:07:00.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.927+0000 7fc2047ba640 1 librados: init done 2024-03-25T18:07:00.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.928+0000 7fc2047ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:00.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.928+0000 7fc2047ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:00.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.928+0000 7fc2047ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:07:00.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:00.928+0000 7fc2047ba640 1 -- 172.21.15.18:0/898030849 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1fc13bee0 con 0x7fc1fc0b1730 2024-03-25T18:07:01.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.212+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1f8001ff0 con 0x7fc1fc0b1730 2024-03-25T18:07:01.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.212+0000 7fc1f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:01.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.212+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc1f8003de0 con 0x7fc1fc0b1730 2024-03-25T18:07:01.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.212+0000 7fc1f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:01.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.212+0000 7fc1f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:01.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.212+0000 7fc1f2ffd640 10 monclient: dump: 2024-03-25T18:07:01.214 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:01.214 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:01.214 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:01.214 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:01.214 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:01.214 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:01.214 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:01.214 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:01.214 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:01.214 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:01.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.212+0000 7fc200d53640 4 set_mon_vals no callback set 2024-03-25T18:07:01.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.213+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fc1f8004000 con 0x7fc1fc0b1730 2024-03-25T18:07:01.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.213+0000 7fc201d55640 10 monclient: get_auth_request con 0x7fc1e006e110 auth_method 0 2024-03-25T18:07:01.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.213+0000 7fc1f2ffd640 10 client.4693.objecter ms_handle_connect 0x7fc1e006e110 2024-03-25T18:07:01.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.218+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 865634137 0 0) 0x7fc1f8054ba0 con 0x7fc1fc0b1730 2024-03-25T18:07:01.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.218+0000 7fc1f2ffd640 10 client.4693.objecter ms_dispatch 0x7fc1fc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:07:01.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.218+0000 7fc1f2ffd640 3 client.4693.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:07:01.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.218+0000 7fc1f2ffd640 3 client.4693.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:07:01.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.218+0000 7fc1f2ffd640 20 client.4693.objecter dump_active .. 0 homeless 2024-03-25T18:07:01.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.218+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc1f80506b0 con 0x7fc1fc0b1730 2024-03-25T18:07:01.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.218+0000 7fc1f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:01.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.218+0000 7fc1f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:01.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.450+0000 7fc2047ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:01.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.450+0000 7fc2047ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:01.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.450+0000 7fc2047ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:07:01.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.450+0000 7fc2047ba640 1 -- 172.21.15.18:0/898030849 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1fc1db140 con 0x7fc1fc0b1730 2024-03-25T18:07:01.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.497+0000 7fc1f2ffd640 1 -- 172.21.15.18:0/898030849 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 678331228) 0x7fc1f80525e0 con 0x7fc1fc0b1730 2024-03-25T18:07:01.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.497+0000 7fc1f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:01.499 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.497+0000 7fc1f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:01.499 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:07:01.499 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T18:06:29.006918+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:06:29.006918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:01.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.500+0000 7fc2047ba640 10 librados: watch_flush enter 2024-03-25T18:07:01.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.500+0000 7fc2047ba640 10 librados: watch_flush exit 2024-03-25T18:07:01.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.500+0000 7fc2047ba640 20 client.4693.objecter shutdown clearing up homeless session... 2024-03-25T18:07:01.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.500+0000 7fc2047ba640 10 client.4693.objecter successfully canceled tick 2024-03-25T18:07:01.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.500+0000 7fc2047ba640 1 -- 172.21.15.18:0/898030849 >> v1:172.21.15.18:6800/3721244451 conn(0x7fc1e006e110 legacy=0x7fc1e00705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:01.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.500+0000 7fc2047ba640 10 monclient: shutdown 2024-03-25T18:07:01.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.500+0000 7fc2047ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:01.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.500+0000 7fc2047ba640 1 -- 172.21.15.18:0/898030849 >> v1:172.21.15.169:6789/0 conn(0x7fc1fc0b1730 legacy=0x7fc1fc1db4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:01.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.500+0000 7fc2047ba640 1 -- 172.21.15.18:0/898030849 shutdown_connections 2024-03-25T18:07:01.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.500+0000 7fc2047ba640 1 -- 172.21.15.18:0/898030849 >> 172.21.15.18:0/898030849 conn(0x7fc1fc0617a0 msgr2=0x7fc1fc137300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:01.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.500+0000 7fc2047ba640 1 -- 172.21.15.18:0/898030849 shutdown_connections 2024-03-25T18:07:01.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.501+0000 7fc2047ba640 1 -- 172.21.15.18:0/898030849 wait complete. 2024-03-25T18:07:01.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:01.501+0000 7fc2047ba640 1 librados: shutdown 2024-03-25T18:07:01.513 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:08.320 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:08.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.439+0000 7fbe3d3e9640 10 monclient: get_monmap_and_config 2024-03-25T18:07:08.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.439+0000 7fbe3d3e9640 10 monclient: build_initial_monmap 2024-03-25T18:07:08.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.439+0000 7fbe3d3e9640 10 monclient: monmap: 2024-03-25T18:07:08.441 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:07:08.441 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:08.441 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:07:08.440704+0000 2024-03-25T18:07:08.441 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:07:08.440704+0000 2024-03-25T18:07:08.441 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:08.441 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:08.441 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:07:08.441 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:07:08.441 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:07:08.441 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:08.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.440+0000 7fbe3d3e9640 1 Processor -- start 2024-03-25T18:07:08.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.440+0000 7fbe3d3e9640 1 -- start start 2024-03-25T18:07:08.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.440+0000 7fbe3d3e9640 10 monclient: init 2024-03-25T18:07:08.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.441+0000 7fbe3d3e9640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:08.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.441+0000 7fbe3d3e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:07:08.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.441+0000 7fbe3d3e9640 10 monclient(hunting): picked mon.noname-b con 0x7fbe3806ab80 addr v1:172.21.15.18:6790/0 2024-03-25T18:07:08.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.441+0000 7fbe3d3e9640 10 monclient(hunting): picked mon.noname-c con 0x7fbe3806af60 addr v1:172.21.15.169:6789/0 2024-03-25T18:07:08.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.441+0000 7fbe3d3e9640 10 monclient(hunting): picked mon.noname-a con 0x7fbe380b00b0 addr v1:172.21.15.18:6789/0 2024-03-25T18:07:08.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.441+0000 7fbe3d3e9640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe380b0500 con 0x7fbe380b00b0 2024-03-25T18:07:08.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.441+0000 7fbe3d3e9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe3813ffa0 con 0x7fbe3806af60 2024-03-25T18:07:08.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.441+0000 7fbe3d3e9640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe380af520 con 0x7fbe3806ab80 2024-03-25T18:07:08.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.441+0000 7fbe3d3e9640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:08.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.441+0000 7fbe3d3e9640 10 monclient(hunting): authenticate will time out at 7557.019818s 2024-03-25T18:07:08.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.446+0000 7fbe36ffd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fbe3806ab80 0x7fbe380ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:49192/0 (socket says 172.21.15.18:49192) 2024-03-25T18:07:08.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.446+0000 7fbe36ffd640 1 -- 172.21.15.18:0/1864285609 learned_addr learned my addr 172.21.15.18:0/1864285609 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:07:08.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.452+0000 7fbe367fc640 1 -- 172.21.15.18:0/1864285609 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906220159 0 0) 0x7fbe380af520 con 0x7fbe3806ab80 2024-03-25T18:07:08.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.452+0000 7fbe367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:08.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.452+0000 7fbe367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:08.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.452+0000 7fbe367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:08.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.452+0000 7fbe367fc640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:07:08.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.452+0000 7fbe367fc640 1 -- 172.21.15.18:0/1864285609 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe24003520 con 0x7fbe3806ab80 2024-03-25T18:07:08.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.468+0000 7fbe367fc640 1 -- 172.21.15.18:0/1864285609 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 235275757 0 0) 0x7fbe24003520 con 0x7fbe3806ab80 2024-03-25T18:07:08.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.468+0000 7fbe367fc640 1 -- 172.21.15.18:0/1864285609 >> v1:172.21.15.169:6789/0 conn(0x7fbe3806af60 legacy=0x7fbe3813f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:08.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.468+0000 7fbe367fc640 1 -- 172.21.15.18:0/1864285609 >> v1:172.21.15.18:6789/0 conn(0x7fbe380b00b0 legacy=0x7fbe3814dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:07:08.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.468+0000 7fbe367fc640 10 monclient: _finish_hunting 0 2024-03-25T18:07:08.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.468+0000 7fbe367fc640 1 monclient: found mon.noname-b 2024-03-25T18:07:08.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.468+0000 7fbe367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:08.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.468+0000 7fbe367fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:07:08.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.468+0000 7fbe367fc640 1 -- 172.21.15.18:0/1864285609 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe38072890 con 0x7fbe3806ab80 2024-03-25T18:07:08.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.468+0000 7fbe367fc640 10 monclient: _finish_auth 0 2024-03-25T18:07:08.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.468+0000 7fbe367fc640 10 monclient: _check_auth_tickets 2024-03-25T18:07:08.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.468+0000 7fbe367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:08.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.468+0000 7fbe367fc640 1 -- 172.21.15.18:0/1864285609 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fbe28001690 con 0x7fbe3806ab80 2024-03-25T18:07:08.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.468+0000 7fbe367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:08.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.468+0000 7fbe367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:08.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.468+0000 7fbe367fc640 10 monclient: dump: 2024-03-25T18:07:08.471 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:08.471 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:08.471 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:08.471 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:08.471 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:08.471 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:08.471 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:08.471 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:08.471 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:08.471 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:08.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.468+0000 7fbe3d3e9640 5 monclient: authenticate success, global_id 4715 2024-03-25T18:07:08.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.468+0000 7fbe3d3e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:08.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.479+0000 7fbe367fc640 1 -- 172.21.15.18:0/1864285609 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe28001860 con 0x7fbe3806ab80 2024-03-25T18:07:08.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.479+0000 7fbe367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:08.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.479+0000 7fbe367fc640 1 -- 172.21.15.18:0/1864285609 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fbe28001cb0 con 0x7fbe3806ab80 2024-03-25T18:07:08.480 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.479+0000 7fbe367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:08.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.479+0000 7fbe367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:08.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.479+0000 7fbe367fc640 10 monclient: dump: 2024-03-25T18:07:08.481 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:08.481 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:08.481 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:08.481 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:08.481 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:08.481 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:08.481 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:08.481 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:08.481 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:08.481 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:08.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.479+0000 7fbe3d3e9640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:08.481 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.479+0000 7fbe3d3e9640 4 set_mon_vals no callback set 2024-03-25T18:07:08.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.480+0000 7fbe3d3e9640 10 monclient: shutdown 2024-03-25T18:07:08.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.480+0000 7fbe3d3e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:08.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.480+0000 7fbe3d3e9640 1 -- 172.21.15.18:0/1864285609 >> v1:172.21.15.18:6790/0 conn(0x7fbe3806ab80 legacy=0x7fbe380ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:08.484 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.480+0000 7fbe3d3e9640 1 -- 172.21.15.18:0/1864285609 shutdown_connections 2024-03-25T18:07:08.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.480+0000 7fbe3d3e9640 1 -- 172.21.15.18:0/1864285609 >> 172.21.15.18:0/1864285609 conn(0x7fbe380b4a80 msgr2=0x7fbe380b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:08.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.480+0000 7fbe3d3e9640 1 -- 172.21.15.18:0/1864285609 shutdown_connections 2024-03-25T18:07:08.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.480+0000 7fbe3d3e9640 1 -- 172.21.15.18:0/1864285609 wait complete. 2024-03-25T18:07:08.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.480+0000 7fbe3d3e9640 10 monclient: build_initial_monmap 2024-03-25T18:07:08.485 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.480+0000 7fbe3d3e9640 10 monclient: monmap: 2024-03-25T18:07:08.485 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:07:08.485 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:08.485 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:07:08.485 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:07:08.485 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:08.485 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:07:08.485 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:07:08.485 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:07:08.485 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:07:08.486 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:08.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.481+0000 7fbe3d3e9640 1 librados: starting msgr at 2024-03-25T18:07:08.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.481+0000 7fbe3d3e9640 1 librados: starting objecter 2024-03-25T18:07:08.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.481+0000 7fbe3d3e9640 1 Processor -- start 2024-03-25T18:07:08.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.481+0000 7fbe3d3e9640 1 -- start start 2024-03-25T18:07:08.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.481+0000 7fbe3d3e9640 1 librados: setting wanted keys 2024-03-25T18:07:08.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.481+0000 7fbe3d3e9640 1 librados: calling monclient init 2024-03-25T18:07:08.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.481+0000 7fbe3d3e9640 10 monclient: init 2024-03-25T18:07:08.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.482+0000 7fbe3d3e9640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:08.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.482+0000 7fbe3d3e9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:07:08.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.482+0000 7fbe3d3e9640 10 monclient(hunting): picked mon.noname-a con 0x7fbe3806ab80 addr v1:172.21.15.18:6789/0 2024-03-25T18:07:08.486 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.482+0000 7fbe3d3e9640 10 monclient(hunting): picked mon.noname-c con 0x7fbe380b00b0 addr v1:172.21.15.18:6790/0 2024-03-25T18:07:08.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.482+0000 7fbe3d3e9640 10 monclient(hunting): picked mon.noname-b con 0x7fbe381dfac0 addr v1:172.21.15.169:6789/0 2024-03-25T18:07:08.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.482+0000 7fbe3d3e9640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe381da740 con 0x7fbe3806ab80 2024-03-25T18:07:08.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.482+0000 7fbe3d3e9640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe38144060 con 0x7fbe381dfac0 2024-03-25T18:07:08.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.482+0000 7fbe3d3e9640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe381db5c0 con 0x7fbe380b00b0 2024-03-25T18:07:08.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.482+0000 7fbe3d3e9640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:08.487 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.482+0000 7fbe3d3e9640 10 monclient(hunting): authenticate will time out at 7557.061144s 2024-03-25T18:07:08.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.485+0000 7fbe2ffff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fbe380b00b0 0x7fbe381daeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:49228/0 (socket says 172.21.15.18:49228) 2024-03-25T18:07:08.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.485+0000 7fbe2ffff640 1 -- 172.21.15.18:0/2019386667 learned_addr learned my addr 172.21.15.18:0/2019386667 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:07:08.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.486+0000 7fbe2f7fe640 10 client.?.objecter ms_handle_connect 0x7fbe380b00b0 2024-03-25T18:07:08.488 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.486+0000 7fbe2f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:08.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.492+0000 7fbe2f7fe640 1 -- 172.21.15.18:0/2019386667 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042731817 0 0) 0x7fbe381db5c0 con 0x7fbe380b00b0 2024-03-25T18:07:08.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.492+0000 7fbe2f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:08.494 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.492+0000 7fbe2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:08.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.492+0000 7fbe2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:08.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.492+0000 7fbe2f7fe640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:07:08.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.492+0000 7fbe2f7fe640 1 -- 172.21.15.18:0/2019386667 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe18004eb0 con 0x7fbe380b00b0 2024-03-25T18:07:08.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.493+0000 7fbe2f7fe640 10 client.?.objecter ms_handle_connect 0x7fbe3806ab80 2024-03-25T18:07:08.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.493+0000 7fbe2f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:08.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.502+0000 7fbe2f7fe640 1 -- 172.21.15.18:0/2019386667 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 582431958 0 0) 0x7fbe18004eb0 con 0x7fbe380b00b0 2024-03-25T18:07:08.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.502+0000 7fbe2f7fe640 1 -- 172.21.15.18:0/2019386667 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe381db5c0 con 0x7fbe380b00b0 2024-03-25T18:07:08.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.502+0000 7fbe2f7fe640 1 -- 172.21.15.18:0/2019386667 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fbe20001a30 con 0x7fbe380b00b0 2024-03-25T18:07:08.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.502+0000 7fbe2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:08.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.502+0000 7fbe2f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:08.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.502+0000 7fbe2f7fe640 10 monclient(hunting): dump: 2024-03-25T18:07:08.504 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:08.504 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:08.504 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:08.504 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:08.504 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:08.504 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:08.504 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:08.505 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:08.505 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:08.505 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:08.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.502+0000 7fbe2f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:08.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.504+0000 7fbe2f7fe640 1 -- 172.21.15.18:0/2019386667 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145506917 0 0) 0x7fbe381da740 con 0x7fbe3806ab80 2024-03-25T18:07:08.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.504+0000 7fbe2f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:08.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.504+0000 7fbe2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:08.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.504+0000 7fbe2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:08.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.504+0000 7fbe2f7fe640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:07:08.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.504+0000 7fbe2f7fe640 1 -- 172.21.15.18:0/2019386667 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe18004eb0 con 0x7fbe3806ab80 2024-03-25T18:07:08.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.504+0000 7fbe2f7fe640 10 client.?.objecter ms_handle_connect 0x7fbe381dfac0 2024-03-25T18:07:08.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.504+0000 7fbe2f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:08.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.508+0000 7fbe2f7fe640 1 -- 172.21.15.18:0/2019386667 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1446928788 0 0) 0x7fbe381db5c0 con 0x7fbe380b00b0 2024-03-25T18:07:08.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.508+0000 7fbe2f7fe640 1 -- 172.21.15.18:0/2019386667 >> v1:172.21.15.169:6789/0 conn(0x7fbe381dfac0 legacy=0x7fbe381e1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:08.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.508+0000 7fbe2f7fe640 1 -- 172.21.15.18:0/2019386667 >> v1:172.21.15.18:6789/0 conn(0x7fbe3806ab80 legacy=0x7fbe38143950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:08.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.508+0000 7fbe2f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:07:08.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.508+0000 7fbe2f7fe640 1 monclient: found mon.c 2024-03-25T18:07:08.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.508+0000 7fbe2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:08.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.508+0000 7fbe2f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:07:08.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.508+0000 7fbe2f7fe640 1 -- 172.21.15.18:0/2019386667 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe38145250 con 0x7fbe380b00b0 2024-03-25T18:07:08.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.508+0000 7fbe2f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:07:08.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.508+0000 7fbe2f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:07:08.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.508+0000 7fbe2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:08.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.509+0000 7fbe3d3e9640 5 monclient: authenticate success, global_id 4718 2024-03-25T18:07:08.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.509+0000 7fbe3d3e9640 10 monclient: _renew_subs 2024-03-25T18:07:08.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.509+0000 7fbe3d3e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:07:08.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.509+0000 7fbe3d3e9640 1 -- 172.21.15.18:0/2019386667 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe381da990 con 0x7fbe380b00b0 2024-03-25T18:07:08.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.509+0000 7fbe3d3e9640 10 client.4718.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:08.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.509+0000 7fbe3d3e9640 10 monclient: _renew_subs 2024-03-25T18:07:08.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.509+0000 7fbe3d3e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:07:08.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.509+0000 7fbe3d3e9640 1 -- 172.21.15.18:0/2019386667 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe38143010 con 0x7fbe380b00b0 2024-03-25T18:07:08.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.509+0000 7fbe3d3e9640 1 librados: init done 2024-03-25T18:07:08.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.510+0000 7fbe3d3e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:08.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.510+0000 7fbe3d3e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:08.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.510+0000 7fbe3d3e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:07:08.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.510+0000 7fbe3d3e9640 1 -- 172.21.15.18:0/2019386667 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe380627b0 con 0x7fbe380b00b0 2024-03-25T18:07:08.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.732+0000 7fbe2f7fe640 1 -- 172.21.15.18:0/2019386667 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe20001cf0 con 0x7fbe380b00b0 2024-03-25T18:07:08.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.732+0000 7fbe2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:08.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.732+0000 7fbe2f7fe640 1 -- 172.21.15.18:0/2019386667 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fbe20003a60 con 0x7fbe380b00b0 2024-03-25T18:07:08.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.732+0000 7fbe2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:08.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.732+0000 7fbe2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:08.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.732+0000 7fbe2f7fe640 10 monclient: dump: 2024-03-25T18:07:08.733 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:08.733 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:08.733 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:08.734 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:08.734 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:08.734 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:08.734 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:08.734 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:08.734 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:08.734 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:08.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.732+0000 7fbe35ffb640 4 set_mon_vals no callback set 2024-03-25T18:07:08.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.822+0000 7fbe2f7fe640 1 -- 172.21.15.18:0/2019386667 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fbe20003bc0 con 0x7fbe380b00b0 2024-03-25T18:07:08.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.823+0000 7fbe36ffd640 10 monclient: get_auth_request con 0x7fbe1806dd10 auth_method 0 2024-03-25T18:07:08.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.823+0000 7fbe2f7fe640 10 client.4718.objecter ms_handle_connect 0x7fbe1806dd10 2024-03-25T18:07:08.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.832+0000 7fbe2f7fe640 1 -- 172.21.15.18:0/2019386667 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 865634137 0 0) 0x7fbe20054910 con 0x7fbe380b00b0 2024-03-25T18:07:08.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.832+0000 7fbe2f7fe640 10 client.4718.objecter ms_dispatch 0x7fbe38000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:07:08.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.832+0000 7fbe2f7fe640 3 client.4718.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:07:08.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.832+0000 7fbe2f7fe640 3 client.4718.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:07:08.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.832+0000 7fbe2f7fe640 20 client.4718.objecter dump_active .. 0 homeless 2024-03-25T18:07:08.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.832+0000 7fbe2f7fe640 1 -- 172.21.15.18:0/2019386667 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe20050250 con 0x7fbe380b00b0 2024-03-25T18:07:08.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.832+0000 7fbe2f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:08.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:08.832+0000 7fbe2f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:09.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.094+0000 7fbe3d3e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:09.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.094+0000 7fbe3d3e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:09.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.094+0000 7fbe3d3e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:07:09.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.094+0000 7fbe3d3e9640 1 -- 172.21.15.18:0/2019386667 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe381dab20 con 0x7fbe380b00b0 2024-03-25T18:07:09.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.245+0000 7fbe2f7fe640 1 -- 172.21.15.18:0/2019386667 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 678331228) 0x7fbe20052350 con 0x7fbe380b00b0 2024-03-25T18:07:09.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.245+0000 7fbe2f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:09.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.245+0000 7fbe2f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:09.247 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:07:09.247 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T18:06:29.006918+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:06:29.006918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:09.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.248+0000 7fbe3d3e9640 10 librados: watch_flush enter 2024-03-25T18:07:09.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.248+0000 7fbe3d3e9640 10 librados: watch_flush exit 2024-03-25T18:07:09.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.248+0000 7fbe3d3e9640 20 client.4718.objecter shutdown clearing up homeless session... 2024-03-25T18:07:09.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.248+0000 7fbe3d3e9640 10 client.4718.objecter successfully canceled tick 2024-03-25T18:07:09.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.248+0000 7fbe3d3e9640 1 -- 172.21.15.18:0/2019386667 >> v1:172.21.15.18:6800/3721244451 conn(0x7fbe1806dd10 legacy=0x7fbe180701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:09.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.248+0000 7fbe3d3e9640 10 monclient: shutdown 2024-03-25T18:07:09.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.248+0000 7fbe3d3e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:09.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.248+0000 7fbe3d3e9640 1 -- 172.21.15.18:0/2019386667 >> v1:172.21.15.18:6790/0 conn(0x7fbe380b00b0 legacy=0x7fbe381daeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:09.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.248+0000 7fbe3d3e9640 1 -- 172.21.15.18:0/2019386667 shutdown_connections 2024-03-25T18:07:09.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.248+0000 7fbe3d3e9640 1 -- 172.21.15.18:0/2019386667 >> 172.21.15.18:0/2019386667 conn(0x7fbe380b4a80 msgr2=0x7fbe380b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:09.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.248+0000 7fbe3d3e9640 1 -- 172.21.15.18:0/2019386667 shutdown_connections 2024-03-25T18:07:09.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.248+0000 7fbe3d3e9640 1 -- 172.21.15.18:0/2019386667 wait complete. 2024-03-25T18:07:09.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:09.248+0000 7fbe3d3e9640 1 librados: shutdown 2024-03-25T18:07:09.262 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:09.273 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:07.066 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:09.320 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:07.069 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:09.321 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:07.070 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:09.322 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:07.115 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:09.332 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:07.080 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:09.363 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:07.156 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:09.400 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:07.192 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:09.404 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:07.197 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:09.434 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:07.226 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:16.067 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:16.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.182+0000 7fd1c9f3d640 10 monclient: get_monmap_and_config 2024-03-25T18:07:16.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.182+0000 7fd1c9f3d640 10 monclient: build_initial_monmap 2024-03-25T18:07:16.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.182+0000 7fd1c9f3d640 10 monclient: monmap: 2024-03-25T18:07:16.184 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:07:16.184 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:16.184 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:07:16.183823+0000 2024-03-25T18:07:16.184 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:07:16.183823+0000 2024-03-25T18:07:16.184 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:16.184 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:16.184 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:07:16.185 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:07:16.185 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:07:16.185 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:16.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.183+0000 7fd1c9f3d640 1 Processor -- start 2024-03-25T18:07:16.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.183+0000 7fd1c9f3d640 1 -- start start 2024-03-25T18:07:16.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.183+0000 7fd1c9f3d640 10 monclient: init 2024-03-25T18:07:16.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.183+0000 7fd1c9f3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:16.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.183+0000 7fd1c9f3d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:07:16.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.183+0000 7fd1c9f3d640 10 monclient(hunting): picked mon.noname-b con 0x7fd1c40b10d0 addr v1:172.21.15.18:6790/0 2024-03-25T18:07:16.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.184+0000 7fd1c9f3d640 10 monclient(hunting): picked mon.noname-c con 0x7fd1c40b4360 addr v1:172.21.15.169:6789/0 2024-03-25T18:07:16.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.184+0000 7fd1c9f3d640 10 monclient(hunting): picked mon.noname-a con 0x7fd1c40a9470 addr v1:172.21.15.18:6789/0 2024-03-25T18:07:16.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.184+0000 7fd1c9f3d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1c40a9ab0 con 0x7fd1c40a9470 2024-03-25T18:07:16.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.184+0000 7fd1c9f3d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1c40b2400 con 0x7fd1c40b4360 2024-03-25T18:07:16.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.184+0000 7fd1c9f3d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1c40aa2a0 con 0x7fd1c40b10d0 2024-03-25T18:07:16.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.184+0000 7fd1c9f3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:16.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.184+0000 7fd1c9f3d640 10 monclient(hunting): authenticate will time out at 7564.762816s 2024-03-25T18:07:16.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.189+0000 7fd1c3fff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fd1c40a9470 0x7fd1c414dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:41890/0 (socket says 172.21.15.18:41890) 2024-03-25T18:07:16.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.189+0000 7fd1c3fff640 1 -- 172.21.15.18:0/4119436199 learned_addr learned my addr 172.21.15.18:0/4119436199 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:07:16.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.206+0000 7fd1c27fc640 1 -- 172.21.15.18:0/4119436199 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062317308 0 0) 0x7fd1c40a9ab0 con 0x7fd1c40a9470 2024-03-25T18:07:16.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.206+0000 7fd1c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:16.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.206+0000 7fd1c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:16.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.206+0000 7fd1c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:16.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.206+0000 7fd1c27fc640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:07:16.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.206+0000 7fd1c27fc640 1 -- 172.21.15.18:0/4119436199 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1ac003520 con 0x7fd1c40a9470 2024-03-25T18:07:16.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.217+0000 7fd1c27fc640 1 -- 172.21.15.18:0/4119436199 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784266776 0 0) 0x7fd1c40b2400 con 0x7fd1c40b4360 2024-03-25T18:07:16.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.217+0000 7fd1c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:16.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.217+0000 7fd1c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:16.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.217+0000 7fd1c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:16.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.217+0000 7fd1c27fc640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:07:16.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.217+0000 7fd1c27fc640 1 -- 172.21.15.18:0/4119436199 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1c40a9ab0 con 0x7fd1c40b4360 2024-03-25T18:07:16.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.221+0000 7fd1c27fc640 1 -- 172.21.15.18:0/4119436199 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4244835576 0 0) 0x7fd1ac003520 con 0x7fd1c40a9470 2024-03-25T18:07:16.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.221+0000 7fd1c27fc640 1 -- 172.21.15.18:0/4119436199 >> v1:172.21.15.18:6790/0 conn(0x7fd1c40b10d0 legacy=0x7fd1c40b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:16.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.221+0000 7fd1c27fc640 1 -- 172.21.15.18:0/4119436199 >> v1:172.21.15.169:6789/0 conn(0x7fd1c40b4360 legacy=0x7fd1c40b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:16.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.221+0000 7fd1c27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:07:16.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.221+0000 7fd1c27fc640 1 monclient: found mon.noname-a 2024-03-25T18:07:16.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.221+0000 7fd1c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:16.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.221+0000 7fd1c27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:07:16.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.221+0000 7fd1c27fc640 1 -- 172.21.15.18:0/4119436199 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c40b14b0 con 0x7fd1c40a9470 2024-03-25T18:07:16.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.221+0000 7fd1c27fc640 10 monclient: _finish_auth 0 2024-03-25T18:07:16.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.221+0000 7fd1c27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:07:16.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.221+0000 7fd1c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:16.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.221+0000 7fd1c27fc640 1 -- 172.21.15.18:0/4119436199 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd1bc001a10 con 0x7fd1c40a9470 2024-03-25T18:07:16.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.221+0000 7fd1c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:16.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.221+0000 7fd1c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:16.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.221+0000 7fd1c27fc640 10 monclient: dump: 2024-03-25T18:07:16.224 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:16.224 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:16.224 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:16.224 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:16.224 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:16.224 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:16.224 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:16.224 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:16.224 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:16.224 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:16.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.221+0000 7fd1c9f3d640 5 monclient: authenticate success, global_id 4713 2024-03-25T18:07:16.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.221+0000 7fd1c9f3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:16.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.231+0000 7fd1c27fc640 1 -- 172.21.15.18:0/4119436199 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1bc001c30 con 0x7fd1c40a9470 2024-03-25T18:07:16.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.231+0000 7fd1c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:16.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.231+0000 7fd1c27fc640 1 -- 172.21.15.18:0/4119436199 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd1bc001fc0 con 0x7fd1c40a9470 2024-03-25T18:07:16.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.231+0000 7fd1c9f3d640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:16.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.231+0000 7fd1c9f3d640 4 set_mon_vals no callback set 2024-03-25T18:07:16.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.231+0000 7fd1c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:16.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.231+0000 7fd1c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:16.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.231+0000 7fd1c27fc640 10 monclient: dump: 2024-03-25T18:07:16.233 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:16.233 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:16.233 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:16.233 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:16.233 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:16.233 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:16.233 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:16.233 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:16.233 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:16.234 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:16.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.232+0000 7fd1c9f3d640 10 monclient: shutdown 2024-03-25T18:07:16.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.232+0000 7fd1c9f3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:16.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.232+0000 7fd1c9f3d640 1 -- 172.21.15.18:0/4119436199 >> v1:172.21.15.18:6789/0 conn(0x7fd1c40a9470 legacy=0x7fd1c414dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:16.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.232+0000 7fd1c9f3d640 1 -- 172.21.15.18:0/4119436199 shutdown_connections 2024-03-25T18:07:16.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.232+0000 7fd1c9f3d640 1 -- 172.21.15.18:0/4119436199 >> 172.21.15.18:0/4119436199 conn(0x7fd1c40617a0 msgr2=0x7fd1c4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:16.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.232+0000 7fd1c9f3d640 1 -- 172.21.15.18:0/4119436199 shutdown_connections 2024-03-25T18:07:16.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.232+0000 7fd1c9f3d640 1 -- 172.21.15.18:0/4119436199 wait complete. 2024-03-25T18:07:16.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.233+0000 7fd1c9f3d640 10 monclient: build_initial_monmap 2024-03-25T18:07:16.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.233+0000 7fd1c9f3d640 10 monclient: monmap: 2024-03-25T18:07:16.235 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:07:16.235 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:16.236 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:07:16.236 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:07:16.236 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:16.236 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:07:16.236 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:07:16.236 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:07:16.236 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:07:16.236 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:16.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.234+0000 7fd1c9f3d640 1 librados: starting msgr at 2024-03-25T18:07:16.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.234+0000 7fd1c9f3d640 1 librados: starting objecter 2024-03-25T18:07:16.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.234+0000 7fd1c9f3d640 1 Processor -- start 2024-03-25T18:07:16.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.234+0000 7fd1c9f3d640 1 -- start start 2024-03-25T18:07:16.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.234+0000 7fd1c9f3d640 1 librados: setting wanted keys 2024-03-25T18:07:16.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.234+0000 7fd1c9f3d640 1 librados: calling monclient init 2024-03-25T18:07:16.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.234+0000 7fd1c9f3d640 10 monclient: init 2024-03-25T18:07:16.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.235+0000 7fd1c9f3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:16.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.235+0000 7fd1c9f3d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:07:16.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.235+0000 7fd1c9f3d640 10 monclient(hunting): picked mon.noname-a con 0x7fd1c40a9470 addr v1:172.21.15.18:6789/0 2024-03-25T18:07:16.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.235+0000 7fd1c9f3d640 10 monclient(hunting): picked mon.noname-b con 0x7fd1c40b10d0 addr v1:172.21.15.169:6789/0 2024-03-25T18:07:16.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.235+0000 7fd1c9f3d640 10 monclient(hunting): picked mon.noname-c con 0x7fd1c40b4360 addr v1:172.21.15.18:6790/0 2024-03-25T18:07:16.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.235+0000 7fd1c9f3d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1c4136690 con 0x7fd1c40a9470 2024-03-25T18:07:16.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.235+0000 7fd1c9f3d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1c4136b20 con 0x7fd1c40b10d0 2024-03-25T18:07:16.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.235+0000 7fd1c9f3d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1c413b4c0 con 0x7fd1c40b4360 2024-03-25T18:07:16.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.235+0000 7fd1c9f3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:16.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.235+0000 7fd1c9f3d640 10 monclient(hunting): authenticate will time out at 7564.814188s 2024-03-25T18:07:16.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.242+0000 7fd1c37fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fd1c40a9470 0x7fd1c41331e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:41932/0 (socket says 172.21.15.18:41932) 2024-03-25T18:07:16.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.242+0000 7fd1c37fe640 1 -- 172.21.15.18:0/2400974256 learned_addr learned my addr 172.21.15.18:0/2400974256 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:07:16.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.242+0000 7fd1abfff640 10 client.?.objecter ms_handle_connect 0x7fd1c40a9470 2024-03-25T18:07:16.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.242+0000 7fd1abfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:16.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.247+0000 7fd1abfff640 10 client.?.objecter ms_handle_connect 0x7fd1c40b10d0 2024-03-25T18:07:16.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.247+0000 7fd1abfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:16.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.251+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197660482 0 0) 0x7fd1c4136690 con 0x7fd1c40a9470 2024-03-25T18:07:16.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.251+0000 7fd1abfff640 10 monclient(hunting): _negotiate 2024-03-25T18:07:16.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.251+0000 7fd1abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:16.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.251+0000 7fd1abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:16.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.251+0000 7fd1abfff640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:07:16.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.251+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1b0004eb0 con 0x7fd1c40a9470 2024-03-25T18:07:16.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.253+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208718512 0 0) 0x7fd1c4136b20 con 0x7fd1c40b10d0 2024-03-25T18:07:16.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.253+0000 7fd1abfff640 10 monclient(hunting): _negotiate 2024-03-25T18:07:16.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.253+0000 7fd1abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:16.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.253+0000 7fd1abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:16.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.253+0000 7fd1abfff640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:07:16.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.253+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1c4136690 con 0x7fd1c40b10d0 2024-03-25T18:07:16.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.257+0000 7fd1abfff640 10 client.?.objecter ms_handle_connect 0x7fd1c40b4360 2024-03-25T18:07:16.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.257+0000 7fd1abfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:16.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.264+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522704306 0 0) 0x7fd1c413b4c0 con 0x7fd1c40b4360 2024-03-25T18:07:16.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.264+0000 7fd1abfff640 10 monclient(hunting): _negotiate 2024-03-25T18:07:16.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.264+0000 7fd1abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:16.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.264+0000 7fd1abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:16.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.264+0000 7fd1abfff640 10 monclient(hunting): my global_id is 4724 2024-03-25T18:07:16.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.264+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1c4136b20 con 0x7fd1c40b4360 2024-03-25T18:07:16.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.265+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1364897037 0 0) 0x7fd1c4136690 con 0x7fd1c40b10d0 2024-03-25T18:07:16.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.265+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1c413b4c0 con 0x7fd1c40b10d0 2024-03-25T18:07:16.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.265+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd1b8001850 con 0x7fd1c40b10d0 2024-03-25T18:07:16.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.265+0000 7fd1abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:16.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.265+0000 7fd1abfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:16.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.265+0000 7fd1abfff640 10 monclient(hunting): dump: 2024-03-25T18:07:16.267 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:16.267 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:16.268 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:16.268 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:16.268 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:16.268 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:16.268 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:16.268 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:16.268 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:16.268 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:16.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.265+0000 7fd1abfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:16.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.271+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1265980492 0 0) 0x7fd1b0004eb0 con 0x7fd1c40a9470 2024-03-25T18:07:16.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.271+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1c4136690 con 0x7fd1c40a9470 2024-03-25T18:07:16.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.271+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd1b4001a30 con 0x7fd1c40a9470 2024-03-25T18:07:16.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.271+0000 7fd1abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:16.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.271+0000 7fd1abfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:16.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.271+0000 7fd1abfff640 10 monclient(hunting): dump: 2024-03-25T18:07:16.273 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:16.273 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:16.273 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:16.274 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:16.274 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:16.274 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:16.274 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:16.274 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:16.274 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:16.274 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:16.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3466071748 0 0) 0x7fd1c413b4c0 con 0x7fd1c40b10d0 2024-03-25T18:07:16.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 >> v1:172.21.15.18:6790/0 conn(0x7fd1c40b4360 legacy=0x7fd1c41e2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:16.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 >> v1:172.21.15.18:6789/0 conn(0x7fd1c40a9470 legacy=0x7fd1c41331e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:16.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1abfff640 10 monclient: _finish_hunting 0 2024-03-25T18:07:16.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1abfff640 1 monclient: found mon.b 2024-03-25T18:07:16.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:16.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1abfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:07:16.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c41360c0 con 0x7fd1c40b10d0 2024-03-25T18:07:16.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1abfff640 10 monclient: _finish_auth 0 2024-03-25T18:07:16.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1abfff640 10 monclient: _check_auth_tickets 2024-03-25T18:07:16.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:16.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1c9f3d640 5 monclient: authenticate success, global_id 4705 2024-03-25T18:07:16.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1c9f3d640 10 monclient: _renew_subs 2024-03-25T18:07:16.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1c9f3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:07:16.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1c9f3d640 1 -- 172.21.15.18:0/2400974256 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c4136370 con 0x7fd1c40b10d0 2024-03-25T18:07:16.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1c9f3d640 10 client.4705.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:16.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1c9f3d640 10 monclient: _renew_subs 2024-03-25T18:07:16.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1c9f3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:07:16.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1c9f3d640 1 -- 172.21.15.18:0/2400974256 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c4135820 con 0x7fd1c40b10d0 2024-03-25T18:07:16.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.275+0000 7fd1c9f3d640 1 librados: init done 2024-03-25T18:07:16.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.276+0000 7fd1c9f3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:16.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.276+0000 7fd1c9f3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:16.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.276+0000 7fd1c9f3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:07:16.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.276+0000 7fd1c9f3d640 1 -- 172.21.15.18:0/2400974256 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c40627b0 con 0x7fd1c40b10d0 2024-03-25T18:07:16.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.567+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1b8001d40 con 0x7fd1c40b10d0 2024-03-25T18:07:16.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.567+0000 7fd1abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:16.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.567+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fd1b80037b0 con 0x7fd1c40b10d0 2024-03-25T18:07:16.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.567+0000 7fd1abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:16.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.567+0000 7fd1abfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:16.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.567+0000 7fd1abfff640 10 monclient: dump: 2024-03-25T18:07:16.569 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:16.569 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:16.569 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:16.569 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:16.570 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:16.570 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:16.570 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:16.570 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:16.570 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:16.570 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:16.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.567+0000 7fd1c1ffb640 4 set_mon_vals no callback set 2024-03-25T18:07:16.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.567+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fd1b8019800 con 0x7fd1c40b10d0 2024-03-25T18:07:16.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.568+0000 7fd1c37fe640 10 monclient: get_auth_request con 0x7fd1b006df70 auth_method 0 2024-03-25T18:07:16.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.568+0000 7fd1abfff640 10 client.4705.objecter ms_handle_connect 0x7fd1b006df70 2024-03-25T18:07:16.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.572+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 865634137 0 0) 0x7fd1b8053550 con 0x7fd1c40b10d0 2024-03-25T18:07:16.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.572+0000 7fd1abfff640 10 client.4705.objecter ms_dispatch 0x7fd1c4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:07:16.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.572+0000 7fd1abfff640 3 client.4705.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:07:16.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.572+0000 7fd1abfff640 3 client.4705.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:07:16.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.572+0000 7fd1abfff640 20 client.4705.objecter dump_active .. 0 homeless 2024-03-25T18:07:16.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.573+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd1b804ee90 con 0x7fd1c40b10d0 2024-03-25T18:07:16.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.573+0000 7fd1abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:16.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.573+0000 7fd1abfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:16.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.820+0000 7fd1c9f3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:16.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.820+0000 7fd1c9f3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:16.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.820+0000 7fd1c9f3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:07:16.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.820+0000 7fd1c9f3d640 1 -- 172.21.15.18:0/2400974256 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1c4136840 con 0x7fd1c40b10d0 2024-03-25T18:07:16.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.848+0000 7fd1abfff640 1 -- 172.21.15.18:0/2400974256 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 678331228) 0x7fd1c4136840 con 0x7fd1c40b10d0 2024-03-25T18:07:16.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.848+0000 7fd1abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:16.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.848+0000 7fd1abfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:16.850 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:07:16.850 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T18:06:29.006918+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:06:29.006918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:16.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.851+0000 7fd1c9f3d640 10 librados: watch_flush enter 2024-03-25T18:07:16.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.851+0000 7fd1c9f3d640 10 librados: watch_flush exit 2024-03-25T18:07:16.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.851+0000 7fd1c9f3d640 20 client.4705.objecter shutdown clearing up homeless session... 2024-03-25T18:07:16.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.851+0000 7fd1c9f3d640 10 client.4705.objecter successfully canceled tick 2024-03-25T18:07:16.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.851+0000 7fd1c9f3d640 1 -- 172.21.15.18:0/2400974256 >> v1:172.21.15.18:6800/3721244451 conn(0x7fd1b006df70 legacy=0x7fd1b0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:16.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.851+0000 7fd1c9f3d640 10 monclient: shutdown 2024-03-25T18:07:16.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.851+0000 7fd1c9f3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:16.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.851+0000 7fd1c9f3d640 1 -- 172.21.15.18:0/2400974256 >> v1:172.21.15.169:6789/0 conn(0x7fd1c40b10d0 legacy=0x7fd1c41342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:16.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.851+0000 7fd1c9f3d640 1 -- 172.21.15.18:0/2400974256 shutdown_connections 2024-03-25T18:07:16.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.851+0000 7fd1c9f3d640 1 -- 172.21.15.18:0/2400974256 >> 172.21.15.18:0/2400974256 conn(0x7fd1c40617a0 msgr2=0x7fd1c40b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:16.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.851+0000 7fd1c9f3d640 1 -- 172.21.15.18:0/2400974256 shutdown_connections 2024-03-25T18:07:16.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.851+0000 7fd1c9f3d640 1 -- 172.21.15.18:0/2400974256 wait complete. 2024-03-25T18:07:16.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:16.851+0000 7fd1c9f3d640 1 librados: shutdown 2024-03-25T18:07:16.863 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:23.669 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:23.787 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.785+0000 7fc2e455d640 10 monclient: get_monmap_and_config 2024-03-25T18:07:23.787 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.785+0000 7fc2e455d640 10 monclient: build_initial_monmap 2024-03-25T18:07:23.787 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.785+0000 7fc2e455d640 10 monclient: monmap: 2024-03-25T18:07:23.787 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:07:23.787 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:23.787 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:07:23.787118+0000 2024-03-25T18:07:23.787 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:07:23.787118+0000 2024-03-25T18:07:23.788 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:23.788 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:23.788 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:07:23.788 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:07:23.788 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:07:23.788 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:23.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.786+0000 7fc2e455d640 1 Processor -- start 2024-03-25T18:07:23.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.786+0000 7fc2e455d640 1 -- start start 2024-03-25T18:07:23.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.786+0000 7fc2e455d640 10 monclient: init 2024-03-25T18:07:23.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.787+0000 7fc2e455d640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:23.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.787+0000 7fc2e455d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:07:23.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.787+0000 7fc2e455d640 10 monclient(hunting): picked mon.noname-b con 0x7fc2dc0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:07:23.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.787+0000 7fc2e455d640 10 monclient(hunting): picked mon.noname-a con 0x7fc2dc13f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:07:23.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.787+0000 7fc2e455d640 10 monclient(hunting): picked mon.noname-c con 0x7fc2dc1430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:07:23.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.787+0000 7fc2e455d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc142520 con 0x7fc2dc13f710 2024-03-25T18:07:23.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.787+0000 7fc2e455d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc0b28a0 con 0x7fc2dc1430c0 2024-03-25T18:07:23.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.787+0000 7fc2e455d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc0b2a30 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.787+0000 7fc2e455d640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:23.790 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.787+0000 7fc2e455d640 10 monclient(hunting): authenticate will time out at 7572.365976s 2024-03-25T18:07:23.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.791+0000 7fc2e22f9640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fc2dc0a98f0 0x7fc2dc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:60656/0 (socket says 172.21.15.18:60656) 2024-03-25T18:07:23.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.791+0000 7fc2e22f9640 1 -- 172.21.15.18:0/2755991462 learned_addr learned my addr 172.21.15.18:0/2755991462 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:07:23.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.801+0000 7fc2e12f7640 1 -- 172.21.15.18:0/2755991462 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001607587 0 0) 0x7fc2dc0b2a30 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.801+0000 7fc2e12f7640 10 monclient(hunting): _negotiate 2024-03-25T18:07:23.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.801+0000 7fc2e12f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:23.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.801+0000 7fc2e12f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:23.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.801+0000 7fc2e12f7640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:07:23.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.801+0000 7fc2e12f7640 1 -- 172.21.15.18:0/2755991462 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c4003520 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.809+0000 7fc2e12f7640 1 -- 172.21.15.18:0/2755991462 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3101467651 0 0) 0x7fc2c4003520 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.809+0000 7fc2e12f7640 1 -- 172.21.15.18:0/2755991462 >> v1:172.21.15.169:6789/0 conn(0x7fc2dc1430c0 legacy=0x7fc2dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:23.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.809+0000 7fc2e12f7640 1 -- 172.21.15.18:0/2755991462 >> v1:172.21.15.18:6789/0 conn(0x7fc2dc13f710 legacy=0x7fc2dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:23.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.809+0000 7fc2e12f7640 10 monclient: _finish_hunting 0 2024-03-25T18:07:23.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.809+0000 7fc2e12f7640 1 monclient: found mon.noname-b 2024-03-25T18:07:23.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.809+0000 7fc2e12f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:23.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.809+0000 7fc2e12f7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:07:23.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.809+0000 7fc2e12f7640 1 -- 172.21.15.18:0/2755991462 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2dc0b44c0 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.809+0000 7fc2e12f7640 10 monclient: _finish_auth 0 2024-03-25T18:07:23.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.809+0000 7fc2e12f7640 10 monclient: _check_auth_tickets 2024-03-25T18:07:23.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.809+0000 7fc2e12f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:23.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.809+0000 7fc2e12f7640 1 -- 172.21.15.18:0/2755991462 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc2cc001a10 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.809+0000 7fc2e12f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:23.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.809+0000 7fc2e12f7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:23.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.809+0000 7fc2e12f7640 10 monclient: dump: 2024-03-25T18:07:23.812 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:23.812 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:23.812 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:23.812 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:23.812 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:23.812 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:23.812 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:23.812 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:23.812 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:23.812 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:23.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.809+0000 7fc2e455d640 5 monclient: authenticate success, global_id 4727 2024-03-25T18:07:23.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.809+0000 7fc2e455d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:23.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.815+0000 7fc2e12f7640 1 -- 172.21.15.18:0/2755991462 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2cc001b70 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.815+0000 7fc2e12f7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:23.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.815+0000 7fc2e12f7640 1 -- 172.21.15.18:0/2755991462 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc2cc001f80 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.815+0000 7fc2e12f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:23.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.815+0000 7fc2e12f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:23.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.815+0000 7fc2e12f7640 10 monclient: dump: 2024-03-25T18:07:23.816 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:23.817 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:23.817 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:23.817 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:23.817 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:23.817 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:23.817 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:23.817 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:23.817 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:23.817 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:23.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.815+0000 7fc2e455d640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:23.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.815+0000 7fc2e455d640 4 set_mon_vals no callback set 2024-03-25T18:07:23.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.815+0000 7fc2e455d640 10 monclient: shutdown 2024-03-25T18:07:23.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.815+0000 7fc2e455d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:23.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.815+0000 7fc2e455d640 1 -- 172.21.15.18:0/2755991462 >> v1:172.21.15.18:6790/0 conn(0x7fc2dc0a98f0 legacy=0x7fc2dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:23.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.816+0000 7fc2e455d640 1 -- 172.21.15.18:0/2755991462 shutdown_connections 2024-03-25T18:07:23.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.816+0000 7fc2e455d640 1 -- 172.21.15.18:0/2755991462 >> 172.21.15.18:0/2755991462 conn(0x7fc2dc0b7ea0 msgr2=0x7fc2dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:23.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.816+0000 7fc2e455d640 1 -- 172.21.15.18:0/2755991462 shutdown_connections 2024-03-25T18:07:23.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.816+0000 7fc2e455d640 1 -- 172.21.15.18:0/2755991462 wait complete. 2024-03-25T18:07:23.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.817+0000 7fc2e455d640 10 monclient: build_initial_monmap 2024-03-25T18:07:23.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.817+0000 7fc2e455d640 10 monclient: monmap: 2024-03-25T18:07:23.819 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:07:23.819 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:23.819 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:07:23.819 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:07:23.820 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:23.820 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:07:23.820 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:07:23.820 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:07:23.820 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:07:23.820 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:23.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.817+0000 7fc2e455d640 1 librados: starting msgr at 2024-03-25T18:07:23.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.817+0000 7fc2e455d640 1 librados: starting objecter 2024-03-25T18:07:23.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.817+0000 7fc2e455d640 1 Processor -- start 2024-03-25T18:07:23.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.817+0000 7fc2e455d640 1 -- start start 2024-03-25T18:07:23.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.817+0000 7fc2e455d640 1 librados: setting wanted keys 2024-03-25T18:07:23.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.817+0000 7fc2e455d640 1 librados: calling monclient init 2024-03-25T18:07:23.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.817+0000 7fc2e455d640 10 monclient: init 2024-03-25T18:07:23.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.818+0000 7fc2e455d640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:23.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.818+0000 7fc2e455d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:07:23.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.818+0000 7fc2e455d640 10 monclient(hunting): picked mon.noname-c con 0x7fc2dc0a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:07:23.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.818+0000 7fc2e455d640 10 monclient(hunting): picked mon.noname-b con 0x7fc2dc13f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:07:23.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.818+0000 7fc2e455d640 10 monclient(hunting): picked mon.noname-a con 0x7fc2dc1430c0 addr v1:172.21.15.18:6789/0 2024-03-25T18:07:23.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.818+0000 7fc2e455d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc135590 con 0x7fc2dc1430c0 2024-03-25T18:07:23.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.818+0000 7fc2e455d640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc136e50 con 0x7fc2dc13f710 2024-03-25T18:07:23.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.818+0000 7fc2e455d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc135070 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.818+0000 7fc2e455d640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:23.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.818+0000 7fc2e455d640 10 monclient(hunting): authenticate will time out at 7572.397458s 2024-03-25T18:07:23.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.827+0000 7fc2e1af8640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fc2dc13f710 0x7fc2dc1d7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:54964/0 (socket says 172.21.15.18:54964) 2024-03-25T18:07:23.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.828+0000 7fc2e1af8640 1 -- 172.21.15.18:0/2887194499 learned_addr learned my addr 172.21.15.18:0/2887194499 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:07:23.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.829+0000 7fc2daffd640 10 client.?.objecter ms_handle_connect 0x7fc2dc13f710 2024-03-25T18:07:23.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.829+0000 7fc2daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:23.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.834+0000 7fc2daffd640 10 client.?.objecter ms_handle_connect 0x7fc2dc1430c0 2024-03-25T18:07:23.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.835+0000 7fc2daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:23.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.839+0000 7fc2daffd640 10 client.?.objecter ms_handle_connect 0x7fc2dc0a98f0 2024-03-25T18:07:23.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.839+0000 7fc2daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:23.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.842+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178546855 0 0) 0x7fc2dc136e50 con 0x7fc2dc13f710 2024-03-25T18:07:23.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.842+0000 7fc2daffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:23.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.842+0000 7fc2daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:23.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.842+0000 7fc2daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:23.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.842+0000 7fc2daffd640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:07:23.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.842+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c8004eb0 con 0x7fc2dc13f710 2024-03-25T18:07:23.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.847+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448215805 0 0) 0x7fc2dc135590 con 0x7fc2dc1430c0 2024-03-25T18:07:23.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.847+0000 7fc2daffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:23.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.847+0000 7fc2daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:23.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.847+0000 7fc2daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:23.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.847+0000 7fc2daffd640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:07:23.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.847+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2dc136e50 con 0x7fc2dc1430c0 2024-03-25T18:07:23.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.861+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249404175 0 0) 0x7fc2dc135070 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.861+0000 7fc2daffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:23.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.861+0000 7fc2daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:23.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.861+0000 7fc2daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:23.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.861+0000 7fc2daffd640 10 monclient(hunting): my global_id is 4730 2024-03-25T18:07:23.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.862+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2dc135590 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.872+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1628182918 0 0) 0x7fc2dc135590 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.872+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2dc135070 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.872+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc2cc001e80 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.872+0000 7fc2daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:23.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.872+0000 7fc2daffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:23.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.872+0000 7fc2daffd640 10 monclient(hunting): dump: 2024-03-25T18:07:23.874 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:23.874 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:23.874 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:23.874 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:23.874 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:23.874 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:23.875 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:23.875 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:23.875 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:23.875 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:23.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.872+0000 7fc2daffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:23.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.874+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1009420712 0 0) 0x7fc2c8004eb0 con 0x7fc2dc13f710 2024-03-25T18:07:23.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.874+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2dc135590 con 0x7fc2dc13f710 2024-03-25T18:07:23.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.874+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc2d0001a30 con 0x7fc2dc13f710 2024-03-25T18:07:23.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.874+0000 7fc2daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:23.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.874+0000 7fc2daffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:23.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.874+0000 7fc2daffd640 10 monclient(hunting): dump: 2024-03-25T18:07:23.876 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:23.876 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:23.876 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:23.876 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:23.876 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:23.876 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:23.877 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:23.877 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:23.877 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:23.877 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:23.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.876+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 806375964 0 0) 0x7fc2dc136e50 con 0x7fc2dc1430c0 2024-03-25T18:07:23.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.876+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2c8004eb0 con 0x7fc2dc1430c0 2024-03-25T18:07:23.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.876+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc2d4001790 con 0x7fc2dc1430c0 2024-03-25T18:07:23.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.876+0000 7fc2daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:23.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.876+0000 7fc2daffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:23.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.876+0000 7fc2daffd640 10 monclient(hunting): dump: 2024-03-25T18:07:23.878 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:23.878 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:23.878 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:23.878 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:23.878 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:23.878 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:23.878 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:23.878 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:23.878 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:23.879 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:23.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2515139807 0 0) 0x7fc2dc135070 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 >> v1:172.21.15.169:6789/0 conn(0x7fc2dc13f710 legacy=0x7fc2dc1d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:23.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 >> v1:172.21.15.18:6789/0 conn(0x7fc2dc1430c0 legacy=0x7fc2dc1e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:23.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2daffd640 10 monclient: _finish_hunting 0 2024-03-25T18:07:23.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2daffd640 1 monclient: found mon.c 2024-03-25T18:07:23.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:23.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2daffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:07:23.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2dc135d80 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2daffd640 10 monclient: _finish_auth 0 2024-03-25T18:07:23.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2daffd640 10 monclient: _check_auth_tickets 2024-03-25T18:07:23.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:23.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2e455d640 5 monclient: authenticate success, global_id 4730 2024-03-25T18:07:23.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2e455d640 10 monclient: _renew_subs 2024-03-25T18:07:23.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2e455d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:07:23.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2e455d640 1 -- 172.21.15.18:0/2887194499 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2dc131660 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2e455d640 10 client.4730.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:23.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2e455d640 10 monclient: _renew_subs 2024-03-25T18:07:23.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2e455d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:07:23.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2e455d640 1 -- 172.21.15.18:0/2887194499 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2dc0b78a0 con 0x7fc2dc0a98f0 2024-03-25T18:07:23.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.904+0000 7fc2e455d640 1 librados: init done 2024-03-25T18:07:23.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.905+0000 7fc2e455d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:23.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.905+0000 7fc2e455d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:23.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.905+0000 7fc2e455d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:07:23.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:23.905+0000 7fc2e455d640 1 -- 172.21.15.18:0/2887194499 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2dc135f70 con 0x7fc2dc0a98f0 2024-03-25T18:07:24.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.232+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2cc001fe0 con 0x7fc2dc0a98f0 2024-03-25T18:07:24.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.232+0000 7fc2daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:24.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.232+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fc2cc004e10 con 0x7fc2dc0a98f0 2024-03-25T18:07:24.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.232+0000 7fc2daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:24.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.232+0000 7fc2daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:24.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.232+0000 7fc2daffd640 10 monclient: dump: 2024-03-25T18:07:24.234 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:24.234 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:24.234 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:24.234 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:24.234 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:24.234 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:24.234 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:24.234 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:24.235 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:24.235 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:24.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.232+0000 7fc2e0af6640 4 set_mon_vals no callback set 2024-03-25T18:07:24.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.233+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fc2cc01ade0 con 0x7fc2dc0a98f0 2024-03-25T18:07:24.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.233+0000 7fc2e1af8640 10 monclient: get_auth_request con 0x7fc2c806e0a0 auth_method 0 2024-03-25T18:07:24.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.233+0000 7fc2daffd640 10 client.4730.objecter ms_handle_connect 0x7fc2c806e0a0 2024-03-25T18:07:24.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.239+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 865634137 0 0) 0x7fc2cc054980 con 0x7fc2dc0a98f0 2024-03-25T18:07:24.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.239+0000 7fc2daffd640 10 client.4730.objecter ms_dispatch 0x7fc2dc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:07:24.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.239+0000 7fc2daffd640 3 client.4730.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:07:24.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.239+0000 7fc2daffd640 3 client.4730.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:07:24.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.239+0000 7fc2daffd640 20 client.4730.objecter dump_active .. 0 homeless 2024-03-25T18:07:24.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.239+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2cc050490 con 0x7fc2dc0a98f0 2024-03-25T18:07:24.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.239+0000 7fc2daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:24.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.239+0000 7fc2daffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:24.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.496+0000 7fc2e455d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:24.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.496+0000 7fc2e455d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:24.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.496+0000 7fc2e455d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:07:24.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.496+0000 7fc2e455d640 1 -- 172.21.15.18:0/2887194499 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2dc135200 con 0x7fc2dc0a98f0 2024-03-25T18:07:24.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.522+0000 7fc2daffd640 1 -- 172.21.15.18:0/2887194499 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 678331228) 0x7fc2cc0523c0 con 0x7fc2dc0a98f0 2024-03-25T18:07:24.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.522+0000 7fc2daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:24.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.522+0000 7fc2daffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:24.524 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:07:24.524 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T18:06:29.006918+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:06:29.006918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:24.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.525+0000 7fc2e455d640 10 librados: watch_flush enter 2024-03-25T18:07:24.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.525+0000 7fc2e455d640 10 librados: watch_flush exit 2024-03-25T18:07:24.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.525+0000 7fc2e455d640 20 client.4730.objecter shutdown clearing up homeless session... 2024-03-25T18:07:24.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.525+0000 7fc2e455d640 10 client.4730.objecter successfully canceled tick 2024-03-25T18:07:24.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.525+0000 7fc2e455d640 1 -- 172.21.15.18:0/2887194499 >> v1:172.21.15.18:6800/3721244451 conn(0x7fc2c806e0a0 legacy=0x7fc2c8070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:24.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.525+0000 7fc2e455d640 10 monclient: shutdown 2024-03-25T18:07:24.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.525+0000 7fc2e455d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:24.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.525+0000 7fc2e455d640 1 -- 172.21.15.18:0/2887194499 >> v1:172.21.15.18:6790/0 conn(0x7fc2dc0a98f0 legacy=0x7fc2dc130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:24.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.525+0000 7fc2e455d640 1 -- 172.21.15.18:0/2887194499 shutdown_connections 2024-03-25T18:07:24.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.525+0000 7fc2e455d640 1 -- 172.21.15.18:0/2887194499 >> 172.21.15.18:0/2887194499 conn(0x7fc2dc0b7ea0 msgr2=0x7fc2dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:24.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.525+0000 7fc2e455d640 1 -- 172.21.15.18:0/2887194499 shutdown_connections 2024-03-25T18:07:24.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.526+0000 7fc2e455d640 1 -- 172.21.15.18:0/2887194499 wait complete. 2024-03-25T18:07:24.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:24.526+0000 7fc2e455d640 1 librados: shutdown 2024-03-25T18:07:24.538 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:29.354 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:27.147 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:29.361 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:27.154 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:29.369 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:27.118 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:29.372 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:27.121 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:29.374 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:27.166 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:29.377 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:27.169 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:29.392 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:27.141 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:29.440 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:27.232 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:29.465 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:27.258 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:29.661 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:27.409 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:29.666 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:27.415 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:29.702 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:27.450 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:29.745 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:27.538 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:29.748 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:27.540 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:29.772 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:27.564 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:29.795 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:27.588 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:29.802 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:27.594 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:29.839 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:27.631 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:31.345 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:31.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.459+0000 7f6635b7a640 10 monclient: get_monmap_and_config 2024-03-25T18:07:31.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.459+0000 7f6635b7a640 10 monclient: build_initial_monmap 2024-03-25T18:07:31.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.459+0000 7f6635b7a640 10 monclient: monmap: 2024-03-25T18:07:31.461 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:07:31.461 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:31.461 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:07:31.460904+0000 2024-03-25T18:07:31.461 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:07:31.460904+0000 2024-03-25T18:07:31.461 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:31.461 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:31.461 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:07:31.461 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:07:31.462 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:07:31.462 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:31.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.460+0000 7f6635b7a640 1 Processor -- start 2024-03-25T18:07:31.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.460+0000 7f6635b7a640 1 -- start start 2024-03-25T18:07:31.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.460+0000 7f6635b7a640 10 monclient: init 2024-03-25T18:07:31.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.461+0000 7f6635b7a640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:31.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.461+0000 7f6635b7a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:07:31.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.461+0000 7f6635b7a640 10 monclient(hunting): picked mon.noname-c con 0x7f66300b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T18:07:31.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.461+0000 7f6635b7a640 10 monclient(hunting): picked mon.noname-b con 0x7f66300b1730 addr v1:172.21.15.18:6790/0 2024-03-25T18:07:31.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.461+0000 7f6635b7a640 10 monclient(hunting): picked mon.noname-a con 0x7f663014be90 addr v1:172.21.15.18:6789/0 2024-03-25T18:07:31.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.461+0000 7f6635b7a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66300af940 con 0x7f663014be90 2024-03-25T18:07:31.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.461+0000 7f6635b7a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66300b0a70 con 0x7f66300b4d60 2024-03-25T18:07:31.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.461+0000 7f6635b7a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66300b0d50 con 0x7f66300b1730 2024-03-25T18:07:31.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.461+0000 7f6635b7a640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:31.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.461+0000 7f6635b7a640 10 monclient(hunting): authenticate will time out at 7580.040084s 2024-03-25T18:07:31.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.476+0000 7f662f7fe640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f66300b4d60 0x7f663013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:54530/0 (socket says 172.21.15.18:54530) 2024-03-25T18:07:31.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.476+0000 7f662f7fe640 1 -- 172.21.15.18:0/2768610543 learned_addr learned my addr 172.21.15.18:0/2768610543 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:07:31.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.500+0000 7f662e7fc640 1 -- 172.21.15.18:0/2768610543 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181815368 0 0) 0x7f66300b0a70 con 0x7f66300b4d60 2024-03-25T18:07:31.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.500+0000 7f662e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:31.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.500+0000 7f662e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:31.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.500+0000 7f662e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:31.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.500+0000 7f662e7fc640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:07:31.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.500+0000 7f662e7fc640 1 -- 172.21.15.18:0/2768610543 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6618003520 con 0x7f66300b4d60 2024-03-25T18:07:31.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.501+0000 7f662e7fc640 1 -- 172.21.15.18:0/2768610543 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203647636 0 0) 0x7f66300af940 con 0x7f663014be90 2024-03-25T18:07:31.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.501+0000 7f662e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:31.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.501+0000 7f662e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:31.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.501+0000 7f662e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:31.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.501+0000 7f662e7fc640 10 monclient(hunting): my global_id is 4725 2024-03-25T18:07:31.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.501+0000 7f662e7fc640 1 -- 172.21.15.18:0/2768610543 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66300b0a70 con 0x7f663014be90 2024-03-25T18:07:31.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.515+0000 7f662e7fc640 1 -- 172.21.15.18:0/2768610543 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057270114 0 0) 0x7f66300b0d50 con 0x7f66300b1730 2024-03-25T18:07:31.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.515+0000 7f662e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:31.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.515+0000 7f662e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:31.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.515+0000 7f662e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:31.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.515+0000 7f662e7fc640 10 monclient(hunting): my global_id is 4733 2024-03-25T18:07:31.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.515+0000 7f662e7fc640 1 -- 172.21.15.18:0/2768610543 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66300af940 con 0x7f66300b1730 2024-03-25T18:07:31.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f662e7fc640 1 -- 172.21.15.18:0/2768610543 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1739805750 0 0) 0x7f6618003520 con 0x7f66300b4d60 2024-03-25T18:07:31.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f662e7fc640 1 -- 172.21.15.18:0/2768610543 >> v1:172.21.15.18:6790/0 conn(0x7f66300b1730 legacy=0x7f66300b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:31.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f662e7fc640 1 -- 172.21.15.18:0/2768610543 >> v1:172.21.15.18:6789/0 conn(0x7f663014be90 legacy=0x7f663014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:31.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f662e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:07:31.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f662e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:07:31.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f662e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:31.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f662e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:07:31.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f662e7fc640 1 -- 172.21.15.18:0/2768610543 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66300b1030 con 0x7f66300b4d60 2024-03-25T18:07:31.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f662e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:07:31.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f662e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:07:31.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f662e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:31.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f662e7fc640 1 -- 172.21.15.18:0/2768610543 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f6620001690 con 0x7f66300b4d60 2024-03-25T18:07:31.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f6635b7a640 5 monclient: authenticate success, global_id 4714 2024-03-25T18:07:31.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f6635b7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:31.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f662e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:31.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f662e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:31.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f662e7fc640 10 monclient: dump: 2024-03-25T18:07:31.519 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:31.519 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:31.519 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:31.519 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:31.519 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:31.519 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:31.519 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:31.519 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:31.519 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:31.519 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:31.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.516+0000 7f6635b7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:31.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.542+0000 7f662e7fc640 1 -- 172.21.15.18:0/2768610543 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66200017f0 con 0x7f66300b4d60 2024-03-25T18:07:31.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.542+0000 7f662e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:31.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.542+0000 7f662e7fc640 1 -- 172.21.15.18:0/2768610543 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f6620001c00 con 0x7f66300b4d60 2024-03-25T18:07:31.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.542+0000 7f6635b7a640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:31.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.542+0000 7f6635b7a640 4 set_mon_vals no callback set 2024-03-25T18:07:31.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.542+0000 7f662e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:31.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.542+0000 7f662e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:31.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.542+0000 7f662e7fc640 10 monclient: dump: 2024-03-25T18:07:31.544 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:31.544 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:31.544 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:31.544 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:31.544 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:31.544 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:31.545 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:31.545 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:31.545 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:31.545 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:31.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.543+0000 7f6635b7a640 10 monclient: shutdown 2024-03-25T18:07:31.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.543+0000 7f6635b7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:31.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.543+0000 7f6635b7a640 1 -- 172.21.15.18:0/2768610543 >> v1:172.21.15.169:6789/0 conn(0x7f66300b4d60 legacy=0x7f663013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:31.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.543+0000 7f6635b7a640 1 -- 172.21.15.18:0/2768610543 shutdown_connections 2024-03-25T18:07:31.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.543+0000 7f6635b7a640 1 -- 172.21.15.18:0/2768610543 >> 172.21.15.18:0/2768610543 conn(0x7f66300617a0 msgr2=0x7f6630061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:31.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.543+0000 7f6635b7a640 1 -- 172.21.15.18:0/2768610543 shutdown_connections 2024-03-25T18:07:31.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.543+0000 7f6635b7a640 1 -- 172.21.15.18:0/2768610543 wait complete. 2024-03-25T18:07:31.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.544+0000 7f6635b7a640 10 monclient: build_initial_monmap 2024-03-25T18:07:31.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.544+0000 7f6635b7a640 10 monclient: monmap: 2024-03-25T18:07:31.547 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:07:31.547 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:31.547 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:07:31.547 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:07:31.547 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:31.547 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:07:31.547 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:07:31.547 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:07:31.547 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:07:31.547 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:31.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.545+0000 7f6635b7a640 1 librados: starting msgr at 2024-03-25T18:07:31.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.545+0000 7f6635b7a640 1 librados: starting objecter 2024-03-25T18:07:31.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.545+0000 7f6635b7a640 1 Processor -- start 2024-03-25T18:07:31.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.545+0000 7f6635b7a640 1 -- start start 2024-03-25T18:07:31.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.545+0000 7f6635b7a640 1 librados: setting wanted keys 2024-03-25T18:07:31.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.545+0000 7f6635b7a640 1 librados: calling monclient init 2024-03-25T18:07:31.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.545+0000 7f6635b7a640 10 monclient: init 2024-03-25T18:07:31.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.546+0000 7f6635b7a640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:31.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.546+0000 7f6635b7a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:07:31.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.546+0000 7f6635b7a640 10 monclient(hunting): picked mon.noname-a con 0x7f66300b1730 addr v1:172.21.15.18:6789/0 2024-03-25T18:07:31.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.546+0000 7f6635b7a640 10 monclient(hunting): picked mon.noname-c con 0x7f66300b4d60 addr v1:172.21.15.18:6790/0 2024-03-25T18:07:31.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.546+0000 7f6635b7a640 10 monclient(hunting): picked mon.noname-b con 0x7f663014be90 addr v1:172.21.15.169:6789/0 2024-03-25T18:07:31.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.546+0000 7f6635b7a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6630140d00 con 0x7f66300b1730 2024-03-25T18:07:31.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.546+0000 7f6635b7a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6630141cc0 con 0x7f663014be90 2024-03-25T18:07:31.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.546+0000 7f6635b7a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66301d8470 con 0x7f66300b4d60 2024-03-25T18:07:31.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.546+0000 7f6635b7a640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:31.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.546+0000 7f6635b7a640 10 monclient(hunting): authenticate will time out at 7580.125606s 2024-03-25T18:07:31.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.557+0000 7f662f7fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f66300b1730 0x7f66301405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:40508/0 (socket says 172.21.15.18:40508) 2024-03-25T18:07:31.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.557+0000 7f662f7fe640 1 -- 172.21.15.18:0/4004778382 learned_addr learned my addr 172.21.15.18:0/4004778382 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:07:31.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.559+0000 7f6617fff640 10 client.?.objecter ms_handle_connect 0x7f66300b1730 2024-03-25T18:07:31.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.559+0000 7f6617fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:31.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.559+0000 7f6617fff640 10 client.?.objecter ms_handle_connect 0x7f66300b4d60 2024-03-25T18:07:31.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.559+0000 7f6617fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:31.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.568+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034593280 0 0) 0x7f6630140d00 con 0x7f66300b1730 2024-03-25T18:07:31.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.568+0000 7f6617fff640 10 monclient(hunting): _negotiate 2024-03-25T18:07:31.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.568+0000 7f6617fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:31.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.568+0000 7f6617fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:31.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.568+0000 7f6617fff640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:07:31.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.568+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f661c004eb0 con 0x7f66300b1730 2024-03-25T18:07:31.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.569+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242337032 0 0) 0x7f66301d8470 con 0x7f66300b4d60 2024-03-25T18:07:31.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.569+0000 7f6617fff640 10 monclient(hunting): _negotiate 2024-03-25T18:07:31.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.569+0000 7f6617fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:31.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.569+0000 7f6617fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:31.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.569+0000 7f6617fff640 10 monclient(hunting): my global_id is 4736 2024-03-25T18:07:31.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.569+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6630140d00 con 0x7f66300b4d60 2024-03-25T18:07:31.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.571+0000 7f6617fff640 10 client.?.objecter ms_handle_connect 0x7f663014be90 2024-03-25T18:07:31.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.571+0000 7f6617fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:31.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.577+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2588956256 0 0) 0x7f661c004eb0 con 0x7f66300b1730 2024-03-25T18:07:31.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.577+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66301d8470 con 0x7f66300b1730 2024-03-25T18:07:31.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.577+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f6620001bc0 con 0x7f66300b1730 2024-03-25T18:07:31.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.577+0000 7f6617fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:31.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.577+0000 7f6617fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:31.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.577+0000 7f6617fff640 10 monclient(hunting): dump: 2024-03-25T18:07:31.579 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:31.579 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:31.579 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:31.579 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:31.579 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:31.579 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:31.579 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:31.579 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:31.579 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:31.579 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:31.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.577+0000 7f6617fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:31.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.582+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512885010 0 0) 0x7f6630141cc0 con 0x7f663014be90 2024-03-25T18:07:31.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.582+0000 7f6617fff640 10 monclient(hunting): _negotiate 2024-03-25T18:07:31.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.582+0000 7f6617fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:31.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.582+0000 7f6617fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:31.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.582+0000 7f6617fff640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:07:31.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.582+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f661c004eb0 con 0x7f663014be90 2024-03-25T18:07:31.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3826910416 0 0) 0x7f6630140d00 con 0x7f66300b4d60 2024-03-25T18:07:31.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6630141cc0 con 0x7f66300b4d60 2024-03-25T18:07:31.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f66240018b0 con 0x7f66300b4d60 2024-03-25T18:07:31.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6617fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:31.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6617fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:31.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6617fff640 10 monclient(hunting): dump: 2024-03-25T18:07:31.589 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:31.590 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:31.590 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:31.590 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:31.590 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:31.590 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:31.590 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:31.590 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:31.590 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:31.590 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:31.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 651600107 0 0) 0x7f66301d8470 con 0x7f66300b1730 2024-03-25T18:07:31.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 >> v1:172.21.15.18:6790/0 conn(0x7f66300b4d60 legacy=0x7f66301d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:31.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 >> v1:172.21.15.169:6789/0 conn(0x7f663014be90 legacy=0x7f6630134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:31.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6617fff640 10 monclient: _finish_hunting 0 2024-03-25T18:07:31.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6617fff640 1 monclient: found mon.a 2024-03-25T18:07:31.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6617fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:31.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6617fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:07:31.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66301d8600 con 0x7f66300b1730 2024-03-25T18:07:31.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6617fff640 10 monclient: _finish_auth 0 2024-03-25T18:07:31.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6617fff640 10 monclient: _check_auth_tickets 2024-03-25T18:07:31.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6617fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:31.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6635b7a640 5 monclient: authenticate success, global_id 4728 2024-03-25T18:07:31.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6635b7a640 10 monclient: _renew_subs 2024-03-25T18:07:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6635b7a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:07:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6635b7a640 1 -- 172.21.15.18:0/4004778382 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f663013b3c0 con 0x7f66300b1730 2024-03-25T18:07:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6635b7a640 10 client.4728.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6635b7a640 10 monclient: _renew_subs 2024-03-25T18:07:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6635b7a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:07:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6635b7a640 1 -- 172.21.15.18:0/4004778382 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66301425a0 con 0x7f66300b1730 2024-03-25T18:07:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.588+0000 7f6635b7a640 1 librados: init done 2024-03-25T18:07:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.589+0000 7f6635b7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.589+0000 7f6635b7a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.589+0000 7f6635b7a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:07:31.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.589+0000 7f6635b7a640 1 -- 172.21.15.18:0/4004778382 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66300b2270 con 0x7f66300b1730 2024-03-25T18:07:31.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.916+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66200011f0 con 0x7f66300b1730 2024-03-25T18:07:31.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.916+0000 7f6617fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:31.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.916+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f6620004cb0 con 0x7f66300b1730 2024-03-25T18:07:31.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.916+0000 7f6617fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:31.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.916+0000 7f6617fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:31.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.916+0000 7f6617fff640 10 monclient: dump: 2024-03-25T18:07:31.918 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:31.918 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:31.918 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:31.918 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:31.918 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:31.918 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:31.918 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:31.918 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:31.919 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:31.919 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:31.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.916+0000 7f662dffb640 4 set_mon_vals no callback set 2024-03-25T18:07:31.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.916+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f662001ad00 con 0x7f66300b1730 2024-03-25T18:07:31.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.917+0000 7f662effd640 10 monclient: get_auth_request con 0x7f661c06df70 auth_method 0 2024-03-25T18:07:31.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.917+0000 7f6617fff640 10 client.4728.objecter ms_handle_connect 0x7f661c06df70 2024-03-25T18:07:31.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.924+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 865634137 0 0) 0x7f6620054990 con 0x7f66300b1730 2024-03-25T18:07:31.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.924+0000 7f6617fff640 10 client.4728.objecter ms_dispatch 0x7f6630000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:07:31.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.924+0000 7f6617fff640 3 client.4728.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:07:31.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.924+0000 7f6617fff640 3 client.4728.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:07:31.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.924+0000 7f6617fff640 20 client.4728.objecter dump_active .. 0 homeless 2024-03-25T18:07:31.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.924+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f66200502d0 con 0x7f66300b1730 2024-03-25T18:07:31.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.924+0000 7f6617fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:31.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:31.924+0000 7f6617fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:32.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.168+0000 7f6635b7a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:32.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.168+0000 7f6635b7a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:32.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.168+0000 7f6635b7a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:07:32.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.168+0000 7f6635b7a640 1 -- 172.21.15.18:0/4004778382 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66300627b0 con 0x7f66300b1730 2024-03-25T18:07:32.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.205+0000 7f6617fff640 1 -- 172.21.15.18:0/4004778382 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 678331228) 0x7f66200523d0 con 0x7f66300b1730 2024-03-25T18:07:32.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.205+0000 7f6617fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:32.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.205+0000 7f6617fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:32.207 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:07:32.207 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T18:06:29.006918+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:06:29.006918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:32.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.208+0000 7f6635b7a640 10 librados: watch_flush enter 2024-03-25T18:07:32.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.208+0000 7f6635b7a640 10 librados: watch_flush exit 2024-03-25T18:07:32.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.208+0000 7f6635b7a640 20 client.4728.objecter shutdown clearing up homeless session... 2024-03-25T18:07:32.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.208+0000 7f6635b7a640 10 client.4728.objecter successfully canceled tick 2024-03-25T18:07:32.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.208+0000 7f6635b7a640 1 -- 172.21.15.18:0/4004778382 >> v1:172.21.15.18:6800/3721244451 conn(0x7f661c06df70 legacy=0x7f661c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:32.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.208+0000 7f6635b7a640 10 monclient: shutdown 2024-03-25T18:07:32.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.208+0000 7f6635b7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:32.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.208+0000 7f6635b7a640 1 -- 172.21.15.18:0/4004778382 >> v1:172.21.15.18:6789/0 conn(0x7f66300b1730 legacy=0x7f66301405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:32.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.208+0000 7f6635b7a640 1 -- 172.21.15.18:0/4004778382 shutdown_connections 2024-03-25T18:07:32.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.208+0000 7f6635b7a640 1 -- 172.21.15.18:0/4004778382 >> 172.21.15.18:0/4004778382 conn(0x7f66300617a0 msgr2=0x7f66300b6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:32.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.209+0000 7f6635b7a640 1 -- 172.21.15.18:0/4004778382 shutdown_connections 2024-03-25T18:07:32.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.209+0000 7f6635b7a640 1 -- 172.21.15.18:0/4004778382 wait complete. 2024-03-25T18:07:32.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:32.209+0000 7f6635b7a640 1 librados: shutdown 2024-03-25T18:07:32.222 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:39.029 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:39.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.142+0000 7faba0c4a640 10 monclient: get_monmap_and_config 2024-03-25T18:07:39.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.142+0000 7faba0c4a640 10 monclient: build_initial_monmap 2024-03-25T18:07:39.143 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.142+0000 7faba0c4a640 10 monclient: monmap: 2024-03-25T18:07:39.143 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:07:39.143 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:39.143 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:07:39.143449+0000 2024-03-25T18:07:39.144 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:07:39.143449+0000 2024-03-25T18:07:39.144 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:39.144 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:39.144 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:07:39.144 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:07:39.144 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:07:39.144 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:39.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.142+0000 7faba0c4a640 1 Processor -- start 2024-03-25T18:07:39.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.142+0000 7faba0c4a640 1 -- start start 2024-03-25T18:07:39.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.142+0000 7faba0c4a640 10 monclient: init 2024-03-25T18:07:39.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.143+0000 7faba0c4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:39.145 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.143+0000 7faba0c4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:07:39.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.143+0000 7faba0c4a640 10 monclient(hunting): picked mon.noname-a con 0x7fab9c0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:07:39.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.143+0000 7faba0c4a640 10 monclient(hunting): picked mon.noname-c con 0x7fab9c13f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:07:39.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.143+0000 7faba0c4a640 10 monclient(hunting): picked mon.noname-b con 0x7fab9c1430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:07:39.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.143+0000 7faba0c4a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab9c142520 con 0x7fab9c0a98f0 2024-03-25T18:07:39.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.143+0000 7faba0c4a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab9c0b28a0 con 0x7fab9c13f710 2024-03-25T18:07:39.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.143+0000 7faba0c4a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab9c0b2a30 con 0x7fab9c1430c0 2024-03-25T18:07:39.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.143+0000 7faba0c4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:39.146 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.143+0000 7faba0c4a640 10 monclient(hunting): authenticate will time out at 7587.722451s 2024-03-25T18:07:39.149 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.147+0000 7fab9ad9d640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fab9c1430c0 0x7fab9c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:57530/0 (socket says 172.21.15.18:57530) 2024-03-25T18:07:39.149 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.147+0000 7fab9ad9d640 1 -- 172.21.15.18:0/21147304 learned_addr learned my addr 172.21.15.18:0/21147304 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:07:39.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.155+0000 7fab99d9b640 1 -- 172.21.15.18:0/21147304 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888340450 0 0) 0x7fab9c0b2a30 con 0x7fab9c1430c0 2024-03-25T18:07:39.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.155+0000 7fab99d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:07:39.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.155+0000 7fab99d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:39.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.155+0000 7fab99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:39.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.155+0000 7fab99d9b640 10 monclient(hunting): my global_id is 4739 2024-03-25T18:07:39.157 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.155+0000 7fab99d9b640 1 -- 172.21.15.18:0/21147304 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab88003520 con 0x7fab9c1430c0 2024-03-25T18:07:39.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7fab99d9b640 1 -- 172.21.15.18:0/21147304 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1663239766 0 0) 0x7fab88003520 con 0x7fab9c1430c0 2024-03-25T18:07:39.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7fab99d9b640 1 -- 172.21.15.18:0/21147304 >> v1:172.21.15.169:6789/0 conn(0x7fab9c13f710 legacy=0x7fab9c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:39.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7fab99d9b640 1 -- 172.21.15.18:0/21147304 >> v1:172.21.15.18:6789/0 conn(0x7fab9c0a98f0 legacy=0x7fab9c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:07:39.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7fab99d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:07:39.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7fab99d9b640 1 monclient: found mon.noname-b 2024-03-25T18:07:39.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7fab99d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:39.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7fab99d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-25T18:07:39.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7fab99d9b640 1 -- 172.21.15.18:0/21147304 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9c0b44c0 con 0x7fab9c1430c0 2024-03-25T18:07:39.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7fab99d9b640 10 monclient: _finish_auth 0 2024-03-25T18:07:39.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7fab99d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:07:39.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7fab99d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:39.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7fab99d9b640 1 -- 172.21.15.18:0/21147304 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fab94001a10 con 0x7fab9c1430c0 2024-03-25T18:07:39.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7faba0c4a640 5 monclient: authenticate success, global_id 4739 2024-03-25T18:07:39.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7faba0c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:39.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7fab99d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:39.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7fab99d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:39.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7fab99d9b640 10 monclient: dump: 2024-03-25T18:07:39.164 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:39.164 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:39.164 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:39.164 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:39.164 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:39.164 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:39.164 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:39.165 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:39.165 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:39.165 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:39.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.161+0000 7faba0c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:39.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.166+0000 7fab99d9b640 1 -- 172.21.15.18:0/21147304 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab94001be0 con 0x7fab9c1430c0 2024-03-25T18:07:39.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.166+0000 7fab99d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:39.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.166+0000 7fab99d9b640 1 -- 172.21.15.18:0/21147304 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fab94002030 con 0x7fab9c1430c0 2024-03-25T18:07:39.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.166+0000 7fab99d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:39.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.166+0000 7fab99d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:39.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.166+0000 7fab99d9b640 10 monclient: dump: 2024-03-25T18:07:39.168 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:39.168 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:39.168 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:39.168 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:39.168 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:39.169 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:39.169 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:39.169 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:39.169 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:39.169 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:39.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.166+0000 7faba0c4a640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:39.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.166+0000 7faba0c4a640 4 set_mon_vals no callback set 2024-03-25T18:07:39.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.167+0000 7faba0c4a640 10 monclient: shutdown 2024-03-25T18:07:39.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.167+0000 7faba0c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:39.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.167+0000 7faba0c4a640 1 -- 172.21.15.18:0/21147304 >> v1:172.21.15.18:6790/0 conn(0x7fab9c1430c0 legacy=0x7fab9c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:39.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.167+0000 7faba0c4a640 1 -- 172.21.15.18:0/21147304 shutdown_connections 2024-03-25T18:07:39.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.167+0000 7faba0c4a640 1 -- 172.21.15.18:0/21147304 >> 172.21.15.18:0/21147304 conn(0x7fab9c0b7ea0 msgr2=0x7fab9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:39.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.167+0000 7faba0c4a640 1 -- 172.21.15.18:0/21147304 shutdown_connections 2024-03-25T18:07:39.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.167+0000 7faba0c4a640 1 -- 172.21.15.18:0/21147304 wait complete. 2024-03-25T18:07:39.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.168+0000 7faba0c4a640 10 monclient: build_initial_monmap 2024-03-25T18:07:39.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.168+0000 7faba0c4a640 10 monclient: monmap: 2024-03-25T18:07:39.171 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:07:39.171 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:39.172 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:07:39.172 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:07:39.172 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:39.172 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:07:39.172 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:07:39.172 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:07:39.172 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:07:39.172 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:39.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.168+0000 7faba0c4a640 1 librados: starting msgr at 2024-03-25T18:07:39.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.168+0000 7faba0c4a640 1 librados: starting objecter 2024-03-25T18:07:39.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.168+0000 7faba0c4a640 1 Processor -- start 2024-03-25T18:07:39.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.168+0000 7faba0c4a640 1 -- start start 2024-03-25T18:07:39.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.168+0000 7faba0c4a640 1 librados: setting wanted keys 2024-03-25T18:07:39.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.168+0000 7faba0c4a640 1 librados: calling monclient init 2024-03-25T18:07:39.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.168+0000 7faba0c4a640 10 monclient: init 2024-03-25T18:07:39.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.169+0000 7faba0c4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:39.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.169+0000 7faba0c4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:07:39.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.169+0000 7faba0c4a640 10 monclient(hunting): picked mon.noname-a con 0x7fab9c0a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:07:39.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.169+0000 7faba0c4a640 10 monclient(hunting): picked mon.noname-c con 0x7fab9c13f710 addr v1:172.21.15.18:6790/0 2024-03-25T18:07:39.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.169+0000 7faba0c4a640 10 monclient(hunting): picked mon.noname-b con 0x7fab9c1430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:07:39.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.169+0000 7faba0c4a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab9c137ce0 con 0x7fab9c0a98f0 2024-03-25T18:07:39.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.169+0000 7faba0c4a640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab9c137980 con 0x7fab9c1430c0 2024-03-25T18:07:39.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.169+0000 7faba0c4a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab9c139620 con 0x7fab9c13f710 2024-03-25T18:07:39.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.169+0000 7faba0c4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:39.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.169+0000 7faba0c4a640 10 monclient(hunting): authenticate will time out at 7587.748268s 2024-03-25T18:07:39.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.173+0000 7fab93fff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fab9c13f710 0x7fab9c133cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:57548/0 (socket says 172.21.15.18:57548) 2024-03-25T18:07:39.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.173+0000 7fab93fff640 1 -- 172.21.15.18:0/3294652005 learned_addr learned my addr 172.21.15.18:0/3294652005 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:07:39.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.174+0000 7fab92ffd640 10 client.?.objecter ms_handle_connect 0x7fab9c13f710 2024-03-25T18:07:39.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.174+0000 7fab92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:39.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.182+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690338428 0 0) 0x7fab9c139620 con 0x7fab9c13f710 2024-03-25T18:07:39.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.182+0000 7fab92ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:39.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.182+0000 7fab92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:39.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.182+0000 7fab92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:39.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.182+0000 7fab92ffd640 10 monclient(hunting): my global_id is 4742 2024-03-25T18:07:39.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.182+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab7c004eb0 con 0x7fab9c13f710 2024-03-25T18:07:39.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.193+0000 7fab92ffd640 10 client.?.objecter ms_handle_connect 0x7fab9c1430c0 2024-03-25T18:07:39.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.193+0000 7fab92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:39.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.194+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 902177175 0 0) 0x7fab7c004eb0 con 0x7fab9c13f710 2024-03-25T18:07:39.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.194+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab9c139620 con 0x7fab9c13f710 2024-03-25T18:07:39.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.194+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fab84001a30 con 0x7fab9c13f710 2024-03-25T18:07:39.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.194+0000 7fab92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:39.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.194+0000 7fab92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:39.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.194+0000 7fab92ffd640 10 monclient(hunting): dump: 2024-03-25T18:07:39.196 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:39.196 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:39.196 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:39.196 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:39.196 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:39.196 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:39.196 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:39.197 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:39.197 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:39.197 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:39.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.194+0000 7fab92ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:39.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.204+0000 7fab92ffd640 10 client.?.objecter ms_handle_connect 0x7fab9c0a98f0 2024-03-25T18:07:39.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.204+0000 7fab92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:39.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.272+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480739285 0 0) 0x7fab9c137ce0 con 0x7fab9c0a98f0 2024-03-25T18:07:39.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.272+0000 7fab92ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:39.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.272+0000 7fab92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:39.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.272+0000 7fab92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:39.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.272+0000 7fab92ffd640 10 monclient(hunting): my global_id is 4731 2024-03-25T18:07:39.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.272+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab7c004eb0 con 0x7fab9c0a98f0 2024-03-25T18:07:39.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.309+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2361857770 0 0) 0x7fab7c004eb0 con 0x7fab9c0a98f0 2024-03-25T18:07:39.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.309+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab9c137ce0 con 0x7fab9c0a98f0 2024-03-25T18:07:39.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.309+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fab8c001790 con 0x7fab9c0a98f0 2024-03-25T18:07:39.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.309+0000 7fab92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:39.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.309+0000 7fab92ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:39.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.309+0000 7fab92ffd640 10 monclient(hunting): dump: 2024-03-25T18:07:39.311 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:39.311 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:39.311 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:39.311 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:39.311 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:39.311 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:39.311 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:39.311 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:39.311 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:39.311 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:39.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.323+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439729560 0 0) 0x7fab9c137980 con 0x7fab9c1430c0 2024-03-25T18:07:39.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.323+0000 7fab92ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:39.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.323+0000 7fab92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:39.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.323+0000 7fab92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:39.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.323+0000 7fab92ffd640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:07:39.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.323+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab7c004eb0 con 0x7fab9c1430c0 2024-03-25T18:07:39.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.337+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4240892741 0 0) 0x7fab9c137ce0 con 0x7fab9c0a98f0 2024-03-25T18:07:39.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.337+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 >> v1:172.21.15.18:6790/0 conn(0x7fab9c13f710 legacy=0x7fab9c133cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:39.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.337+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 >> v1:172.21.15.169:6789/0 conn(0x7fab9c1430c0 legacy=0x7fab9c1343d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:39.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.337+0000 7fab92ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:07:39.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.337+0000 7fab92ffd640 1 monclient: found mon.a 2024-03-25T18:07:39.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.337+0000 7fab92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:39.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.337+0000 7fab92ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:07:39.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.337+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9c134ae0 con 0x7fab9c0a98f0 2024-03-25T18:07:39.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.337+0000 7fab92ffd640 10 monclient: _finish_auth 0 2024-03-25T18:07:39.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.337+0000 7fab92ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:07:39.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.337+0000 7fab92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:39.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.337+0000 7faba0c4a640 5 monclient: authenticate success, global_id 4731 2024-03-25T18:07:39.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.337+0000 7faba0c4a640 10 monclient: _renew_subs 2024-03-25T18:07:39.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.337+0000 7faba0c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:07:39.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.337+0000 7faba0c4a640 1 -- 172.21.15.18:0/3294652005 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab9c138cf0 con 0x7fab9c0a98f0 2024-03-25T18:07:39.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.338+0000 7faba0c4a640 10 client.4731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:39.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.338+0000 7faba0c4a640 10 monclient: _renew_subs 2024-03-25T18:07:39.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.338+0000 7faba0c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:07:39.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.338+0000 7faba0c4a640 1 -- 172.21.15.18:0/3294652005 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab9c0abac0 con 0x7fab9c0a98f0 2024-03-25T18:07:39.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.338+0000 7faba0c4a640 1 librados: init done 2024-03-25T18:07:39.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.338+0000 7faba0c4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:39.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.338+0000 7faba0c4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:39.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.338+0000 7faba0c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:07:39.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.338+0000 7faba0c4a640 1 -- 172.21.15.18:0/3294652005 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9c0b3ef0 con 0x7fab9c0a98f0 2024-03-25T18:07:39.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.633+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab8c001c80 con 0x7fab9c0a98f0 2024-03-25T18:07:39.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.633+0000 7fab92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:39.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.633+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fab8c003790 con 0x7fab9c0a98f0 2024-03-25T18:07:39.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.633+0000 7fab92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:39.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.633+0000 7fab92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:39.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.633+0000 7fab92ffd640 10 monclient: dump: 2024-03-25T18:07:39.635 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:39.635 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:39.635 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:39.635 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:39.636 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:39.636 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:39.636 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:39.636 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:39.636 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:39.636 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:39.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.633+0000 7fab9959a640 4 set_mon_vals no callback set 2024-03-25T18:07:39.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.634+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fab8c019760 con 0x7fab9c0a98f0 2024-03-25T18:07:39.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.634+0000 7fab93fff640 10 monclient: get_auth_request con 0x7fab7c06def0 auth_method 0 2024-03-25T18:07:39.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.634+0000 7fab92ffd640 10 client.4731.objecter ms_handle_connect 0x7fab7c06def0 2024-03-25T18:07:39.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.639+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 865634137 0 0) 0x7fab8c053430 con 0x7fab9c0a98f0 2024-03-25T18:07:39.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.639+0000 7fab92ffd640 10 client.4731.objecter ms_dispatch 0x7fab9c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:07:39.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.639+0000 7fab92ffd640 3 client.4731.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:07:39.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.639+0000 7fab92ffd640 3 client.4731.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:07:39.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.639+0000 7fab92ffd640 20 client.4731.objecter dump_active .. 0 homeless 2024-03-25T18:07:39.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.640+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fab8c04ed70 con 0x7fab9c0a98f0 2024-03-25T18:07:39.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.640+0000 7fab92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:39.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.640+0000 7fab92ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:39.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.878+0000 7faba0c4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:39.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.878+0000 7faba0c4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:39.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.878+0000 7faba0c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-25T18:07:39.880 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.878+0000 7faba0c4a640 1 -- 172.21.15.18:0/3294652005 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab9c0627b0 con 0x7fab9c0a98f0 2024-03-25T18:07:39.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.915+0000 7fab92ffd640 1 -- 172.21.15.18:0/3294652005 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 678331228) 0x7fab8c050e70 con 0x7fab9c0a98f0 2024-03-25T18:07:39.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.915+0000 7fab92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:39.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.915+0000 7fab92ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:39.917 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:07:39.917 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T18:06:29.006918+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:06:29.006918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:39.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.918+0000 7faba0c4a640 10 librados: watch_flush enter 2024-03-25T18:07:39.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.918+0000 7faba0c4a640 10 librados: watch_flush exit 2024-03-25T18:07:39.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.918+0000 7faba0c4a640 20 client.4731.objecter shutdown clearing up homeless session... 2024-03-25T18:07:39.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.918+0000 7faba0c4a640 10 client.4731.objecter successfully canceled tick 2024-03-25T18:07:39.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.918+0000 7faba0c4a640 1 -- 172.21.15.18:0/3294652005 >> v1:172.21.15.18:6800/3721244451 conn(0x7fab7c06def0 legacy=0x7fab7c0703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:39.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.918+0000 7faba0c4a640 10 monclient: shutdown 2024-03-25T18:07:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.918+0000 7faba0c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.918+0000 7faba0c4a640 1 -- 172.21.15.18:0/3294652005 >> v1:172.21.15.18:6789/0 conn(0x7fab9c0a98f0 legacy=0x7fab9c1daed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.918+0000 7faba0c4a640 1 -- 172.21.15.18:0/3294652005 shutdown_connections 2024-03-25T18:07:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.918+0000 7faba0c4a640 1 -- 172.21.15.18:0/3294652005 >> 172.21.15.18:0/3294652005 conn(0x7fab9c0b7ea0 msgr2=0x7fab9c13c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.918+0000 7faba0c4a640 1 -- 172.21.15.18:0/3294652005 shutdown_connections 2024-03-25T18:07:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.918+0000 7faba0c4a640 1 -- 172.21.15.18:0/3294652005 wait complete. 2024-03-25T18:07:39.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:39.918+0000 7faba0c4a640 1 librados: shutdown 2024-03-25T18:07:39.931 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:46.737 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:46.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.846+0000 7f79d6e96640 10 monclient: get_monmap_and_config 2024-03-25T18:07:46.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.846+0000 7f79d6e96640 10 monclient: build_initial_monmap 2024-03-25T18:07:46.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.846+0000 7f79d6e96640 10 monclient: monmap: 2024-03-25T18:07:46.848 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:07:46.848 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:46.848 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:07:46.848263+0000 2024-03-25T18:07:46.848 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:07:46.848263+0000 2024-03-25T18:07:46.848 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:46.848 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:46.848 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:07:46.849 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:07:46.849 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:07:46.849 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:46.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.847+0000 7f79d6e96640 1 Processor -- start 2024-03-25T18:07:46.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.847+0000 7f79d6e96640 1 -- start start 2024-03-25T18:07:46.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.847+0000 7f79d6e96640 10 monclient: init 2024-03-25T18:07:46.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.848+0000 7f79d6e96640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:46.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.848+0000 7f79d6e96640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:07:46.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.848+0000 7f79d6e96640 10 monclient(hunting): picked mon.noname-a con 0x7f79d00a98f0 addr v1:172.21.15.18:6789/0 2024-03-25T18:07:46.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.848+0000 7f79d6e96640 10 monclient(hunting): picked mon.noname-c con 0x7f79d013f710 addr v1:172.21.15.169:6789/0 2024-03-25T18:07:46.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.848+0000 7f79d6e96640 10 monclient(hunting): picked mon.noname-b con 0x7f79d01430c0 addr v1:172.21.15.18:6790/0 2024-03-25T18:07:46.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.848+0000 7f79d6e96640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d0142520 con 0x7f79d00a98f0 2024-03-25T18:07:46.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.848+0000 7f79d6e96640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d00b28a0 con 0x7f79d013f710 2024-03-25T18:07:46.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.848+0000 7f79d6e96640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d00b2a30 con 0x7f79d01430c0 2024-03-25T18:07:46.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.848+0000 7f79d6e96640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:46.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.848+0000 7f79d6e96640 10 monclient(hunting): authenticate will time out at 7595.427184s 2024-03-25T18:07:46.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.854+0000 7f79cffff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f79d013f710 0x7f79d01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:53724/0 (socket says 172.21.15.18:53724) 2024-03-25T18:07:46.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.854+0000 7f79cffff640 1 -- 172.21.15.18:0/1116358072 learned_addr learned my addr 172.21.15.18:0/1116358072 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:07:46.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.863+0000 7f79d5433640 1 -- 172.21.15.18:0/1116358072 >> v1:172.21.15.18:6790/0 conn(0x7f79d01430c0 legacy=0x7f79d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-25T18:07:46.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.863+0000 7f79d5433640 1 -- 172.21.15.18:0/1116358072 >> v1:172.21.15.18:6790/0 conn(0x7f79d01430c0 legacy=0x7f79d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:07:46.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.863+0000 7f79d5433640 1 --1- 172.21.15.18:0/1116358072 >> v1:172.21.15.18:6790/0 conn(0x7f79d01430c0 0x7f79d014dc30 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:07:46.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.868+0000 7f79cf7fe640 1 -- 172.21.15.18:0/1116358072 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986356865 0 0) 0x7f79d00b28a0 con 0x7f79d013f710 2024-03-25T18:07:46.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.868+0000 7f79cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:46.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.868+0000 7f79cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:46.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.868+0000 7f79cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:46.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.868+0000 7f79cf7fe640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:07:46.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.868+0000 7f79cf7fe640 1 -- 172.21.15.18:0/1116358072 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79b8003520 con 0x7f79d013f710 2024-03-25T18:07:46.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79cf7fe640 1 -- 172.21.15.18:0/1116358072 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 345964401 0 0) 0x7f79b8003520 con 0x7f79d013f710 2024-03-25T18:07:46.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79cf7fe640 1 -- 172.21.15.18:0/1116358072 >> v1:172.21.15.18:6790/0 conn(0x7f79d01430c0 legacy=0x7f79d014dc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:07:46.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79cf7fe640 1 -- 172.21.15.18:0/1116358072 >> v1:172.21.15.18:6789/0 conn(0x7f79d00a98f0 legacy=0x7f79d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:46.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79cf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:07:46.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79cf7fe640 1 monclient: found mon.noname-c 2024-03-25T18:07:46.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:46.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79cf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.169:6789/0 2024-03-25T18:07:46.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79cf7fe640 1 -- 172.21.15.18:0/1116358072 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d00b44c0 con 0x7f79d013f710 2024-03-25T18:07:46.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79cf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:07:46.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79cf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:07:46.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:46.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79cf7fe640 1 -- 172.21.15.18:0/1116358072 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f79c4001690 con 0x7f79d013f710 2024-03-25T18:07:46.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79d6e96640 5 monclient: authenticate success, global_id 4726 2024-03-25T18:07:46.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79d6e96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:46.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79cf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:46.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79cf7fe640 10 monclient: dump: 2024-03-25T18:07:46.885 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:46.885 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:46.886 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:46.886 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:46.886 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.886 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:46.886 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:46.886 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:46.886 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:46.886 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:46.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.882+0000 7f79d6e96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:46.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.897+0000 7f79cf7fe640 1 -- 172.21.15.18:0/1116358072 <== mon.1 v1:172.21.15.169:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79c4001860 con 0x7f79d013f710 2024-03-25T18:07:46.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.897+0000 7f79cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:46.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.897+0000 7f79cf7fe640 1 -- 172.21.15.18:0/1116358072 <== mon.1 v1:172.21.15.169:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f79c4001cb0 con 0x7f79d013f710 2024-03-25T18:07:46.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.897+0000 7f79cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.897+0000 7f79cf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:46.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.897+0000 7f79cf7fe640 10 monclient: dump: 2024-03-25T18:07:46.899 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:46.899 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:46.899 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:46.900 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:46.900 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.900 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:46.900 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:46.900 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:46.900 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:46.900 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:46.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.897+0000 7f79d6e96640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:46.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.897+0000 7f79d6e96640 4 set_mon_vals no callback set 2024-03-25T18:07:46.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.898+0000 7f79d6e96640 10 monclient: shutdown 2024-03-25T18:07:46.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.898+0000 7f79d6e96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:46.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.898+0000 7f79d6e96640 1 -- 172.21.15.18:0/1116358072 >> v1:172.21.15.169:6789/0 conn(0x7f79d013f710 legacy=0x7f79d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:46.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.898+0000 7f79d6e96640 1 -- 172.21.15.18:0/1116358072 shutdown_connections 2024-03-25T18:07:46.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.898+0000 7f79d6e96640 1 -- 172.21.15.18:0/1116358072 >> 172.21.15.18:0/1116358072 conn(0x7f79d00b7ea0 msgr2=0x7f79d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:46.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.898+0000 7f79d6e96640 1 -- 172.21.15.18:0/1116358072 shutdown_connections 2024-03-25T18:07:46.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.898+0000 7f79d6e96640 1 -- 172.21.15.18:0/1116358072 wait complete. 2024-03-25T18:07:46.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.899+0000 7f79d6e96640 10 monclient: build_initial_monmap 2024-03-25T18:07:46.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.899+0000 7f79d6e96640 10 monclient: monmap: 2024-03-25T18:07:46.903 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:07:46.903 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:46.903 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:07:46.903 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:07:46.903 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:46.903 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:07:46.903 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:07:46.903 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:07:46.903 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:07:46.904 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:46.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.899+0000 7f79d6e96640 1 librados: starting msgr at 2024-03-25T18:07:46.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.899+0000 7f79d6e96640 1 librados: starting objecter 2024-03-25T18:07:46.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.899+0000 7f79d6e96640 1 Processor -- start 2024-03-25T18:07:46.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.900+0000 7f79d6e96640 1 -- start start 2024-03-25T18:07:46.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.900+0000 7f79d6e96640 1 librados: setting wanted keys 2024-03-25T18:07:46.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.900+0000 7f79d6e96640 1 librados: calling monclient init 2024-03-25T18:07:46.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.900+0000 7f79d6e96640 10 monclient: init 2024-03-25T18:07:46.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.900+0000 7f79d6e96640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:46.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.900+0000 7f79d6e96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:07:46.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.900+0000 7f79d6e96640 10 monclient(hunting): picked mon.noname-c con 0x7f79d00a98f0 addr v1:172.21.15.18:6790/0 2024-03-25T18:07:46.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.900+0000 7f79d6e96640 10 monclient(hunting): picked mon.noname-a con 0x7f79d013f710 addr v1:172.21.15.18:6789/0 2024-03-25T18:07:46.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.900+0000 7f79d6e96640 10 monclient(hunting): picked mon.noname-b con 0x7f79d01430c0 addr v1:172.21.15.169:6789/0 2024-03-25T18:07:46.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.900+0000 7f79d6e96640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d00abc20 con 0x7f79d013f710 2024-03-25T18:07:46.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.900+0000 7f79d6e96640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d00accf0 con 0x7f79d01430c0 2024-03-25T18:07:46.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.900+0000 7f79d6e96640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d00abe70 con 0x7f79d00a98f0 2024-03-25T18:07:46.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.900+0000 7f79d6e96640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:46.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.900+0000 7f79d6e96640 10 monclient(hunting): authenticate will time out at 7595.479535s 2024-03-25T18:07:46.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.910+0000 7f79d5433640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7f79d01430c0 0x7f79d01e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:53730/0 (socket says 172.21.15.18:53730) 2024-03-25T18:07:46.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.910+0000 7f79d5433640 1 -- 172.21.15.18:0/739821580 learned_addr learned my addr 172.21.15.18:0/739821580 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:07:46.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.911+0000 7f79cd7fa640 10 client.?.objecter ms_handle_connect 0x7f79d01430c0 2024-03-25T18:07:46.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.911+0000 7f79cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:46.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.912+0000 7f79cd7fa640 10 client.?.objecter ms_handle_connect 0x7f79d013f710 2024-03-25T18:07:46.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.912+0000 7f79cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:46.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.918+0000 7f79cd7fa640 10 client.?.objecter ms_handle_connect 0x7f79d00a98f0 2024-03-25T18:07:46.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.918+0000 7f79cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:46.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.919+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941908692 0 0) 0x7f79d00accf0 con 0x7f79d01430c0 2024-03-25T18:07:46.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.919+0000 7f79cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:07:46.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.919+0000 7f79cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:46.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.919+0000 7f79cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:46.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.919+0000 7f79cd7fa640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:07:46.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.919+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79bc004eb0 con 0x7f79d01430c0 2024-03-25T18:07:46.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.924+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808452321 0 0) 0x7f79d00abc20 con 0x7f79d013f710 2024-03-25T18:07:46.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.924+0000 7f79cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:07:46.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.924+0000 7f79cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:46.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.924+0000 7f79cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:46.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.924+0000 7f79cd7fa640 10 monclient(hunting): my global_id is 4734 2024-03-25T18:07:46.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.924+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79d00accf0 con 0x7f79d013f710 2024-03-25T18:07:46.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.929+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823833359 0 0) 0x7f79d00abe70 con 0x7f79d00a98f0 2024-03-25T18:07:46.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.929+0000 7f79cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:07:46.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.929+0000 7f79cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:46.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.929+0000 7f79cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:46.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.929+0000 7f79cd7fa640 10 monclient(hunting): my global_id is 4745 2024-03-25T18:07:46.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.929+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79d00abc20 con 0x7f79d00a98f0 2024-03-25T18:07:46.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.937+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2533196231 0 0) 0x7f79bc004eb0 con 0x7f79d01430c0 2024-03-25T18:07:46.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.937+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79d00abe70 con 0x7f79d01430c0 2024-03-25T18:07:46.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.937+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f79c8004310 con 0x7f79d01430c0 2024-03-25T18:07:46.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.937+0000 7f79cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.937+0000 7f79cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:46.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.937+0000 7f79cd7fa640 10 monclient(hunting): dump: 2024-03-25T18:07:46.939 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:46.939 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:46.939 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:46.939 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:46.939 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.939 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:46.939 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:46.939 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:46.939 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:46.939 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:46.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.937+0000 7f79cd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:46.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.946+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1987691634 0 0) 0x7f79d00abc20 con 0x7f79d00a98f0 2024-03-25T18:07:46.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.946+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79bc004eb0 con 0x7f79d00a98f0 2024-03-25T18:07:46.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.946+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f79c0001830 con 0x7f79d00a98f0 2024-03-25T18:07:46.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.946+0000 7f79cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.946+0000 7f79cd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:46.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.946+0000 7f79cd7fa640 10 monclient(hunting): dump: 2024-03-25T18:07:46.948 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:46.948 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:46.948 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:46.948 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:46.948 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.948 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:46.948 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:46.948 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:46.949 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:46.949 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.947+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2076666078 0 0) 0x7f79d00accf0 con 0x7f79d013f710 2024-03-25T18:07:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.947+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79d00abc20 con 0x7f79d013f710 2024-03-25T18:07:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.947+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f79c4001b70 con 0x7f79d013f710 2024-03-25T18:07:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.947+0000 7f79cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.947+0000 7f79cd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.947+0000 7f79cd7fa640 10 monclient(hunting): dump: 2024-03-25T18:07:46.950 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:46.950 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:46.950 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:46.950 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:46.950 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.950 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:46.950 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:46.950 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:46.950 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:46.950 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 <== mon.1 v1:172.21.15.169:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2780274904 0 0) 0x7f79d00abe70 con 0x7f79d01430c0 2024-03-25T18:07:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 >> v1:172.21.15.18:6790/0 conn(0x7f79d00a98f0 legacy=0x7f79d0138a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 >> v1:172.21.15.18:6789/0 conn(0x7f79d013f710 legacy=0x7f79d00ac5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79cd7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:07:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79cd7fa640 1 monclient: found mon.b 2024-03-25T18:07:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79cd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:07:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 --> v1:172.21.15.169:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d00ac150 con 0x7f79d01430c0 2024-03-25T18:07:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79cd7fa640 10 monclient: _finish_auth 0 2024-03-25T18:07:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79cd7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:07:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79d6e96640 5 monclient: authenticate success, global_id 4729 2024-03-25T18:07:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79d6e96640 10 monclient: _renew_subs 2024-03-25T18:07:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79d6e96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:07:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79d6e96640 1 -- 172.21.15.18:0/739821580 --> v1:172.21.15.169:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d0139280 con 0x7f79d01430c0 2024-03-25T18:07:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79d6e96640 10 client.4729.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79d6e96640 10 monclient: _renew_subs 2024-03-25T18:07:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79d6e96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:07:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79d6e96640 1 -- 172.21.15.18:0/739821580 --> v1:172.21.15.169:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d0138030 con 0x7f79d01430c0 2024-03-25T18:07:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.962+0000 7f79d6e96640 1 librados: init done 2024-03-25T18:07:46.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.963+0000 7f79d6e96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:46.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.963+0000 7f79d6e96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:46.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.963+0000 7f79d6e96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:07:46.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:46.963+0000 7f79d6e96640 1 -- 172.21.15.18:0/739821580 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d00b3ef0 con 0x7f79d01430c0 2024-03-25T18:07:47.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.260+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 <== mon.1 v1:172.21.15.169:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79c8004820 con 0x7f79d01430c0 2024-03-25T18:07:47.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.260+0000 7f79cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:47.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.260+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 <== mon.1 v1:172.21.15.169:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7f79c80013f0 con 0x7f79d01430c0 2024-03-25T18:07:47.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.260+0000 7f79cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:47.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.260+0000 7f79cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:47.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.260+0000 7f79cd7fa640 10 monclient: dump: 2024-03-25T18:07:47.262 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:47.262 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:47.262 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:47.263 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:47.263 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:47.263 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:47.263 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:47.263 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:47.263 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:47.263 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:47.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.260+0000 7f79ceffd640 4 set_mon_vals no callback set 2024-03-25T18:07:47.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.261+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 <== mon.1 v1:172.21.15.169:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7f79c8001610 con 0x7f79d01430c0 2024-03-25T18:07:47.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.262+0000 7f79d4c32640 10 monclient: get_auth_request con 0x7f79bc06e0f0 auth_method 0 2024-03-25T18:07:47.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.262+0000 7f79cd7fa640 10 client.4729.objecter ms_handle_connect 0x7f79bc06e0f0 2024-03-25T18:07:47.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.266+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 <== mon.1 v1:172.21.15.169:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 865634137 0 0) 0x7f79c8054a90 con 0x7f79d01430c0 2024-03-25T18:07:47.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.266+0000 7f79cd7fa640 10 client.4729.objecter ms_dispatch 0x7f79d0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:07:47.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.266+0000 7f79cd7fa640 3 client.4729.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:07:47.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.266+0000 7f79cd7fa640 3 client.4729.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:07:47.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.266+0000 7f79cd7fa640 20 client.4729.objecter dump_active .. 0 homeless 2024-03-25T18:07:47.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.266+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 <== mon.1 v1:172.21.15.169:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f79c80505a0 con 0x7f79d01430c0 2024-03-25T18:07:47.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.266+0000 7f79cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:47.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.266+0000 7f79cd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:47.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.488+0000 7f79d6e96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:47.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.488+0000 7f79d6e96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:47.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.488+0000 7f79d6e96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.169:6789/0 2024-03-25T18:07:47.489 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.488+0000 7f79d6e96640 1 -- 172.21.15.18:0/739821580 --> v1:172.21.15.169:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79d00627b0 con 0x7f79d01430c0 2024-03-25T18:07:47.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.589+0000 7f79cd7fa640 1 -- 172.21.15.18:0/739821580 <== mon.1 v1:172.21.15.169:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 678331228) 0x7f79c80524d0 con 0x7f79d01430c0 2024-03-25T18:07:47.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.589+0000 7f79cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:47.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.589+0000 7f79cd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:47.591 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:07:47.591 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T18:06:29.006918+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:06:29.006918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:47.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.592+0000 7f79d6e96640 10 librados: watch_flush enter 2024-03-25T18:07:47.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.592+0000 7f79d6e96640 10 librados: watch_flush exit 2024-03-25T18:07:47.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.592+0000 7f79d6e96640 20 client.4729.objecter shutdown clearing up homeless session... 2024-03-25T18:07:47.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.592+0000 7f79d6e96640 10 client.4729.objecter successfully canceled tick 2024-03-25T18:07:47.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.592+0000 7f79d6e96640 1 -- 172.21.15.18:0/739821580 >> v1:172.21.15.18:6800/3721244451 conn(0x7f79bc06e0f0 legacy=0x7f79bc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:47.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.592+0000 7f79d6e96640 10 monclient: shutdown 2024-03-25T18:07:47.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.592+0000 7f79d6e96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:47.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.592+0000 7f79d6e96640 1 -- 172.21.15.18:0/739821580 >> v1:172.21.15.169:6789/0 conn(0x7f79d01430c0 legacy=0x7f79d01e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:47.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.592+0000 7f79d6e96640 1 -- 172.21.15.18:0/739821580 shutdown_connections 2024-03-25T18:07:47.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.592+0000 7f79d6e96640 1 -- 172.21.15.18:0/739821580 >> 172.21.15.18:0/739821580 conn(0x7f79d00b7ea0 msgr2=0x7f79d013d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:47.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.592+0000 7f79d6e96640 1 -- 172.21.15.18:0/739821580 shutdown_connections 2024-03-25T18:07:47.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.592+0000 7f79d6e96640 1 -- 172.21.15.18:0/739821580 wait complete. 2024-03-25T18:07:47.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:47.592+0000 7f79d6e96640 1 librados: shutdown 2024-03-25T18:07:47.605 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:54.411 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:54.476 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:52.269 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:54.484 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:52.276 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:54.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.502+0000 7fea1a26e640 10 monclient: get_monmap_and_config 2024-03-25T18:07:54.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.502+0000 7fea1a26e640 10 monclient: build_initial_monmap 2024-03-25T18:07:54.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.502+0000 7fea1a26e640 10 monclient: monmap: 2024-03-25T18:07:54.504 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:07:54.504 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:54.504 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T18:07:54.503939+0000 2024-03-25T18:07:54.504 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T18:07:54.503939+0000 2024-03-25T18:07:54.504 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:54.504 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:54.504 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:07:54.504 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-c 2024-03-25T18:07:54.504 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-25T18:07:54.504 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:54.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.503+0000 7fea1a26e640 1 Processor -- start 2024-03-25T18:07:54.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.503+0000 7fea1a26e640 1 -- start start 2024-03-25T18:07:54.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.503+0000 7fea1a26e640 10 monclient: init 2024-03-25T18:07:54.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.503+0000 7fea1a26e640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:54.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.503+0000 7fea1a26e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:07:54.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.503+0000 7fea1a26e640 10 monclient(hunting): picked mon.noname-c con 0x7fea140b4d60 addr v1:172.21.15.169:6789/0 2024-03-25T18:07:54.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.503+0000 7fea1a26e640 10 monclient(hunting): picked mon.noname-a con 0x7fea140b1730 addr v1:172.21.15.18:6789/0 2024-03-25T18:07:54.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.503+0000 7fea1a26e640 10 monclient(hunting): picked mon.noname-b con 0x7fea1414be90 addr v1:172.21.15.18:6790/0 2024-03-25T18:07:54.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.503+0000 7fea1a26e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea140af940 con 0x7fea140b1730 2024-03-25T18:07:54.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.503+0000 7fea1a26e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea140b0a70 con 0x7fea140b4d60 2024-03-25T18:07:54.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.503+0000 7fea1a26e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea140b0d50 con 0x7fea1414be90 2024-03-25T18:07:54.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.503+0000 7fea1a26e640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:54.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.503+0000 7fea1a26e640 10 monclient(hunting): authenticate will time out at 7603.082594s 2024-03-25T18:07:54.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.519+0000 7fea137fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fea140b1730 0x7fea140b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:41578/0 (socket says 172.21.15.18:41578) 2024-03-25T18:07:54.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.519+0000 7fea137fe640 1 -- 172.21.15.18:0/2817190838 learned_addr learned my addr 172.21.15.18:0/2817190838 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:07:54.528 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:11:52.321 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:54.536 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:52.285 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:54.541 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:52.290 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:54.546 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:52.338 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:54.550 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:52.343 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:54.565 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:11:52.314 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:54.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.565+0000 7fea12ffd640 1 -- 172.21.15.18:0/2817190838 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53302573 0 0) 0x7fea140af940 con 0x7fea140b1730 2024-03-25T18:07:54.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.565+0000 7fea12ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:54.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.565+0000 7fea12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:54.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.565+0000 7fea12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:54.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.565+0000 7fea12ffd640 10 monclient(hunting): my global_id is 4737 2024-03-25T18:07:54.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.565+0000 7fea12ffd640 1 -- 172.21.15.18:0/2817190838 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9fc003520 con 0x7fea140b1730 2024-03-25T18:07:54.577 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:11:52.370 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:54.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.610+0000 7fea12ffd640 1 -- 172.21.15.18:0/2817190838 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142849445 0 0) 0x7fea140b0a70 con 0x7fea140b4d60 2024-03-25T18:07:54.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.610+0000 7fea12ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:54.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.610+0000 7fea12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:54.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.610+0000 7fea12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:54.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.610+0000 7fea12ffd640 10 monclient(hunting): my global_id is 4732 2024-03-25T18:07:54.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.610+0000 7fea12ffd640 1 -- 172.21.15.18:0/2817190838 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea140af940 con 0x7fea140b4d60 2024-03-25T18:07:54.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.618+0000 7fea12ffd640 1 -- 172.21.15.18:0/2817190838 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789969289 0 0) 0x7fea140b0d50 con 0x7fea1414be90 2024-03-25T18:07:54.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.618+0000 7fea12ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:54.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.618+0000 7fea12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:54.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.618+0000 7fea12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:54.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.618+0000 7fea12ffd640 10 monclient(hunting): my global_id is 4748 2024-03-25T18:07:54.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.618+0000 7fea12ffd640 1 -- 172.21.15.18:0/2817190838 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea140b0a70 con 0x7fea1414be90 2024-03-25T18:07:54.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.635+0000 7fea12ffd640 1 -- 172.21.15.18:0/2817190838 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3753356235 0 0) 0x7fe9fc003520 con 0x7fea140b1730 2024-03-25T18:07:54.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.635+0000 7fea12ffd640 1 -- 172.21.15.18:0/2817190838 >> v1:172.21.15.18:6790/0 conn(0x7fea1414be90 legacy=0x7fea1414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:54.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.635+0000 7fea12ffd640 1 -- 172.21.15.18:0/2817190838 >> v1:172.21.15.169:6789/0 conn(0x7fea140b4d60 legacy=0x7fea1413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:54.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.635+0000 7fea12ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:07:54.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.635+0000 7fea12ffd640 1 monclient: found mon.noname-a 2024-03-25T18:07:54.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.635+0000 7fea12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:54.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.635+0000 7fea12ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-25T18:07:54.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.635+0000 7fea12ffd640 1 -- 172.21.15.18:0/2817190838 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea140b1030 con 0x7fea140b1730 2024-03-25T18:07:54.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.635+0000 7fea12ffd640 10 monclient: _finish_auth 0 2024-03-25T18:07:54.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.635+0000 7fea12ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:07:54.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.635+0000 7fea12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:54.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.635+0000 7fea12ffd640 1 -- 172.21.15.18:0/2817190838 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fea08001690 con 0x7fea140b1730 2024-03-25T18:07:54.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.635+0000 7fea12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:54.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.635+0000 7fea12ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:54.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.635+0000 7fea12ffd640 10 monclient: dump: 2024-03-25T18:07:54.638 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:54.638 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:54.638 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:54.638 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:54.638 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:54.638 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:54.638 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:54.638 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:54.638 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:54.638 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:54.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.635+0000 7fea1a26e640 5 monclient: authenticate success, global_id 4737 2024-03-25T18:07:54.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.635+0000 7fea1a26e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:54.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.653+0000 7fea12ffd640 1 -- 172.21.15.18:0/2817190838 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea080017f0 con 0x7fea140b1730 2024-03-25T18:07:54.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.653+0000 7fea12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:54.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.653+0000 7fea12ffd640 1 -- 172.21.15.18:0/2817190838 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fea08001c00 con 0x7fea140b1730 2024-03-25T18:07:54.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.653+0000 7fea12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:54.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.653+0000 7fea12ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:54.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.653+0000 7fea12ffd640 10 monclient: dump: 2024-03-25T18:07:54.655 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:54.655 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:54.655 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:54.655 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:54.655 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:54.655 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:54.655 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:54.655 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:54.655 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:54.655 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:54.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.653+0000 7fea1a26e640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:54.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.653+0000 7fea1a26e640 4 set_mon_vals no callback set 2024-03-25T18:07:54.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.654+0000 7fea1a26e640 10 monclient: shutdown 2024-03-25T18:07:54.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.654+0000 7fea1a26e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:54.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.654+0000 7fea1a26e640 1 -- 172.21.15.18:0/2817190838 >> v1:172.21.15.18:6789/0 conn(0x7fea140b1730 legacy=0x7fea140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:54.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.654+0000 7fea1a26e640 1 -- 172.21.15.18:0/2817190838 shutdown_connections 2024-03-25T18:07:54.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.654+0000 7fea1a26e640 1 -- 172.21.15.18:0/2817190838 >> 172.21.15.18:0/2817190838 conn(0x7fea140617a0 msgr2=0x7fea14061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:54.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.654+0000 7fea1a26e640 1 -- 172.21.15.18:0/2817190838 shutdown_connections 2024-03-25T18:07:54.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.654+0000 7fea1a26e640 1 -- 172.21.15.18:0/2817190838 wait complete. 2024-03-25T18:07:54.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.655+0000 7fea1a26e640 10 monclient: build_initial_monmap 2024-03-25T18:07:54.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.655+0000 7fea1a26e640 10 monclient: monmap: 2024-03-25T18:07:54.657 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-25T18:07:54.657 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:54.657 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-25T18:07:54.657 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-25T18:07:54.657 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:54.657 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-25T18:07:54.657 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-25T18:07:54.657 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.noname-b 2024-03-25T18:07:54.658 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-25T18:07:54.658 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:54.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.656+0000 7fea1a26e640 1 librados: starting msgr at 2024-03-25T18:07:54.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.656+0000 7fea1a26e640 1 librados: starting objecter 2024-03-25T18:07:54.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.656+0000 7fea1a26e640 1 Processor -- start 2024-03-25T18:07:54.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.656+0000 7fea1a26e640 1 -- start start 2024-03-25T18:07:54.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.656+0000 7fea1a26e640 1 librados: setting wanted keys 2024-03-25T18:07:54.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.656+0000 7fea1a26e640 1 librados: calling monclient init 2024-03-25T18:07:54.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.656+0000 7fea1a26e640 10 monclient: init 2024-03-25T18:07:54.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.657+0000 7fea1a26e640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:54.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.657+0000 7fea1a26e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:07:54.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.657+0000 7fea1a26e640 10 monclient(hunting): picked mon.noname-b con 0x7fea140b1730 addr v1:172.21.15.169:6789/0 2024-03-25T18:07:54.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.657+0000 7fea1a26e640 10 monclient(hunting): picked mon.noname-a con 0x7fea140b4d60 addr v1:172.21.15.18:6789/0 2024-03-25T18:07:54.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.657+0000 7fea1a26e640 10 monclient(hunting): picked mon.noname-c con 0x7fea1414be90 addr v1:172.21.15.18:6790/0 2024-03-25T18:07:54.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.657+0000 7fea1a26e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea14144d40 con 0x7fea140b4d60 2024-03-25T18:07:54.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.657+0000 7fea1a26e640 1 -- --> v1:172.21.15.169:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea141dbac0 con 0x7fea140b1730 2024-03-25T18:07:54.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.657+0000 7fea1a26e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea14135d30 con 0x7fea1414be90 2024-03-25T18:07:54.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.657+0000 7fea1a26e640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:54.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.657+0000 7fea1a26e640 10 monclient(hunting): authenticate will time out at 7603.235893s 2024-03-25T18:07:54.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.693+0000 7fea13fff640 1 --1- >> v1:172.21.15.169:6789/0 conn(0x7fea140b1730 0x7fea141db3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.169:6789/0 says I am v1:172.21.15.18:43222/0 (socket says 172.21.15.18:43222) 2024-03-25T18:07:54.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.694+0000 7fea13fff640 1 -- 172.21.15.18:0/592345645 learned_addr learned my addr 172.21.15.18:0/592345645 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-25T18:07:54.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.702+0000 7fea10ff9640 10 client.?.objecter ms_handle_connect 0x7fea1414be90 2024-03-25T18:07:54.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.702+0000 7fea10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:54.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.704+0000 7fea10ff9640 10 client.?.objecter ms_handle_connect 0x7fea140b4d60 2024-03-25T18:07:54.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.704+0000 7fea10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:54.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.710+0000 7fea10ff9640 10 client.?.objecter ms_handle_connect 0x7fea140b1730 2024-03-25T18:07:54.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.710+0000 7fea10ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:54.718 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.717+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521009607 0 0) 0x7fea14135d30 con 0x7fea1414be90 2024-03-25T18:07:54.718 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.717+0000 7fea10ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:07:54.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.717+0000 7fea10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:54.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.717+0000 7fea10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:54.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.717+0000 7fea10ff9640 10 monclient(hunting): my global_id is 4751 2024-03-25T18:07:54.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.717+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea00004eb0 con 0x7fea1414be90 2024-03-25T18:07:54.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.723+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 <== mon.1 v1:172.21.15.169:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257767919 0 0) 0x7fea141dbac0 con 0x7fea140b1730 2024-03-25T18:07:54.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.723+0000 7fea10ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:07:54.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.723+0000 7fea10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:54.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.723+0000 7fea10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:54.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.723+0000 7fea10ff9640 10 monclient(hunting): my global_id is 4735 2024-03-25T18:07:54.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.723+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 --> v1:172.21.15.169:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea14135d30 con 0x7fea140b1730 2024-03-25T18:07:54.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.726+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123448828 0 0) 0x7fea14144d40 con 0x7fea140b4d60 2024-03-25T18:07:54.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.726+0000 7fea10ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:07:54.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.726+0000 7fea10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:54.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.726+0000 7fea10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:54.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.726+0000 7fea10ff9640 10 monclient(hunting): my global_id is 4740 2024-03-25T18:07:54.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.727+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea141dbac0 con 0x7fea140b4d60 2024-03-25T18:07:54.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.741+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 885180262 0 0) 0x7fea00004eb0 con 0x7fea1414be90 2024-03-25T18:07:54.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.741+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea14144d40 con 0x7fea1414be90 2024-03-25T18:07:54.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.741+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fea0c001bd0 con 0x7fea1414be90 2024-03-25T18:07:54.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.741+0000 7fea10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:54.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.741+0000 7fea10ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:54.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.741+0000 7fea10ff9640 10 monclient(hunting): dump: 2024-03-25T18:07:54.743 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:54.743 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:54.744 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:54.744 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:54.744 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:54.744 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:54.744 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:54.744 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:54.744 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:54.744 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:54.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.741+0000 7fea10ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:54.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.749+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 <== mon.1 v1:172.21.15.169:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 655751894 0 0) 0x7fea14135d30 con 0x7fea140b1730 2024-03-25T18:07:54.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.749+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 --> v1:172.21.15.169:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea00004eb0 con 0x7fea140b1730 2024-03-25T18:07:54.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.749+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 <== mon.1 v1:172.21.15.169:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fea040019b0 con 0x7fea140b1730 2024-03-25T18:07:54.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.749+0000 7fea10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:54.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.749+0000 7fea10ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:54.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.749+0000 7fea10ff9640 10 monclient(hunting): dump: 2024-03-25T18:07:54.751 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:54.751 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:54.751 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:54.751 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:54.751 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:54.751 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:54.751 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:54.751 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:54.751 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:54.751 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:54.769 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.768+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1106373890 0 0) 0x7fea141dbac0 con 0x7fea140b4d60 2024-03-25T18:07:54.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.768+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea14135d30 con 0x7fea140b4d60 2024-03-25T18:07:54.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.768+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fea08001b00 con 0x7fea140b4d60 2024-03-25T18:07:54.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.768+0000 7fea10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:54.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.768+0000 7fea10ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:54.770 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.768+0000 7fea10ff9640 10 monclient(hunting): dump: 2024-03-25T18:07:54.770 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:54.770 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:54.770 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:54.770 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:54.770 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:54.770 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:54.770 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:54.770 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:54.770 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:54.770 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:54.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.774+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2884015824 0 0) 0x7fea14144d40 con 0x7fea1414be90 2024-03-25T18:07:54.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.774+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 >> v1:172.21.15.169:6789/0 conn(0x7fea140b1730 legacy=0x7fea141db3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:54.776 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.774+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 >> v1:172.21.15.18:6789/0 conn(0x7fea140b4d60 legacy=0x7fea14144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:54.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.774+0000 7fea10ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:07:54.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.774+0000 7fea10ff9640 1 monclient: found mon.c 2024-03-25T18:07:54.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.774+0000 7fea10ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:54.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.774+0000 7fea10ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:07:54.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.774+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea141382f0 con 0x7fea1414be90 2024-03-25T18:07:54.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.774+0000 7fea10ff9640 10 monclient: _finish_auth 0 2024-03-25T18:07:54.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.774+0000 7fea10ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:07:54.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.774+0000 7fea10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:54.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.774+0000 7fea1a26e640 5 monclient: authenticate success, global_id 4751 2024-03-25T18:07:54.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.774+0000 7fea1a26e640 10 monclient: _renew_subs 2024-03-25T18:07:54.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.774+0000 7fea1a26e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:07:54.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.774+0000 7fea1a26e640 1 -- 172.21.15.18:0/592345645 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea14145670 con 0x7fea1414be90 2024-03-25T18:07:54.777 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.775+0000 7fea1a26e640 10 client.4751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:54.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.775+0000 7fea1a26e640 10 monclient: _renew_subs 2024-03-25T18:07:54.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.775+0000 7fea1a26e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:07:54.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.775+0000 7fea1a26e640 1 -- 172.21.15.18:0/592345645 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea14145920 con 0x7fea1414be90 2024-03-25T18:07:54.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.775+0000 7fea1a26e640 1 librados: init done 2024-03-25T18:07:54.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.776+0000 7fea1a26e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:54.778 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.776+0000 7fea1a26e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:54.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.776+0000 7fea1a26e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:07:54.779 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:54.776+0000 7fea1a26e640 1 -- 172.21.15.18:0/592345645 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea141448e0 con 0x7fea1414be90 2024-03-25T18:07:55.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.093+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea0c002060 con 0x7fea1414be90 2024-03-25T18:07:55.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.093+0000 7fea10ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:55.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.093+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1581560456 0 0) 0x7fea0c003990 con 0x7fea1414be90 2024-03-25T18:07:55.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.093+0000 7fea10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:55.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.093+0000 7fea10ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:55.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.093+0000 7fea10ff9640 10 monclient: dump: 2024-03-25T18:07:55.095 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-25T18:07:55.095 INFO:teuthology.misc.health.smithi018.stderr:fsid c6c98ae9-f2e3-42f0-a1fb-d587f9832636 2024-03-25T18:07:55.095 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:55.095 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-25T17:55:31.237133+0000 2024-03-25T18:07:55.095 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:55.095 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 3 2024-03-25T18:07:55.095 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-25T18:07:55.095 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.169:6789/0 mon.b 2024-03-25T18:07:55.096 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-25T18:07:55.096 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-25T18:07:55.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.093+0000 7fea127fc640 4 set_mon_vals no callback set 2024-03-25T18:07:55.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.094+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3821466333 0 0) 0x7fea0c019960 con 0x7fea1414be90 2024-03-25T18:07:55.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.094+0000 7fea13fff640 10 monclient: get_auth_request con 0x7fea0006e0d0 auth_method 0 2024-03-25T18:07:55.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.094+0000 7fea10ff9640 10 client.4751.objecter ms_handle_connect 0x7fea0006e0d0 2024-03-25T18:07:55.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.103+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 865634137 0 0) 0x7fea0c053680 con 0x7fea1414be90 2024-03-25T18:07:55.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.103+0000 7fea10ff9640 10 client.4751.objecter ms_dispatch 0x7fea14000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:07:55.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.103+0000 7fea10ff9640 3 client.4751.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:07:55.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.103+0000 7fea10ff9640 3 client.4751.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:07:55.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.103+0000 7fea10ff9640 20 client.4751.objecter dump_active .. 0 homeless 2024-03-25T18:07:55.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.103+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea0c04f190 con 0x7fea1414be90 2024-03-25T18:07:55.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.103+0000 7fea10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:55.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.103+0000 7fea10ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:55.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.372+0000 7fea1a26e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:55.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.372+0000 7fea1a26e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:55.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.372+0000 7fea1a26e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-25T18:07:55.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.372+0000 7fea1a26e640 1 -- 172.21.15.18:0/592345645 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea140627b0 con 0x7fea1414be90 2024-03-25T18:07:55.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.396+0000 7fea10ff9640 1 -- 172.21.15.18:0/592345645 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 678331228) 0x7fea0c0510c0 con 0x7fea1414be90 2024-03-25T18:07:55.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.396+0000 7fea10ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:55.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.396+0000 7fea10ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:55.398 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-25T18:07:55.398 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"c6c98ae9-f2e3-42f0-a1fb-d587f9832636","created":"2024-03-25T17:56:24.794278+0000","modified":"2024-03-25T18:06:29.006918+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:06:29.006918+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4f033a33-dc7d-4490-8b69-56479f880977","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"2f60b59e-0fb3-4ad1-a470-9eec2498bb28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"87509107-33d5-45a2-84a2-48a1d184e4b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"d91f737b-db6a-4b15-ad8c-b8a546d3536c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"43877ab5-09f4-45fd-a3bc-bbb38bcb1c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6bfd7ba5-daf8-493c-9ae1-ff6c9d46c4fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c3bd1d77-7390-4452-8980-b0ef85bc74a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7a735ce9-3492-4d28-afa8-13314d24c12a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:55.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.398+0000 7fea1a26e640 10 librados: watch_flush enter 2024-03-25T18:07:55.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.398+0000 7fea1a26e640 10 librados: watch_flush exit 2024-03-25T18:07:55.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.398+0000 7fea1a26e640 20 client.4751.objecter shutdown clearing up homeless session... 2024-03-25T18:07:55.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.398+0000 7fea1a26e640 10 client.4751.objecter successfully canceled tick 2024-03-25T18:07:55.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.398+0000 7fea1a26e640 1 -- 172.21.15.18:0/592345645 >> v1:172.21.15.18:6800/3721244451 conn(0x7fea0006e0d0 legacy=0x7fea00070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:55.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.398+0000 7fea1a26e640 10 monclient: shutdown 2024-03-25T18:07:55.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.398+0000 7fea1a26e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:55.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.398+0000 7fea1a26e640 1 -- 172.21.15.18:0/592345645 >> v1:172.21.15.18:6790/0 conn(0x7fea1414be90 legacy=0x7fea141e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:55.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.398+0000 7fea1a26e640 1 -- 172.21.15.18:0/592345645 shutdown_connections 2024-03-25T18:07:55.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.398+0000 7fea1a26e640 1 -- 172.21.15.18:0/592345645 >> 172.21.15.18:0/592345645 conn(0x7fea140617a0 msgr2=0x7fea140b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:55.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.398+0000 7fea1a26e640 1 -- 172.21.15.18:0/592345645 shutdown_connections 2024-03-25T18:07:55.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.398+0000 7fea1a26e640 1 -- 172.21.15.18:0/592345645 wait complete. 2024-03-25T18:07:55.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-25T18:07:55.398+0000 7fea1a26e640 1 librados: shutdown 2024-03-25T18:07:55.411 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:55.411 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:07:55.413 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:07:55.413 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:07:55.413 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:07:57.301 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:11:26.049 35477== 2024-03-25T18:07:57.301 INFO:tasks.ceph.osd.0.smithi018.stderr:==00:00:11:26.049 35477== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:07:57.341 INFO:tasks.ceph.osd.0.smithi018.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:08:01.515 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:08:01.515 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi018 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:08:01.516 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:08:01.516 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:08:01.516 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:08:03.362 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:11:32.114 35476== 2024-03-25T18:08:03.363 INFO:tasks.ceph.osd.1.smithi018.stderr:==00:00:11:32.114 35476== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:08:03.400 INFO:tasks.ceph.osd.1.smithi018.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:08:07.617 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:08:07.618 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi018 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:08:07.618 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:08:07.618 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:08:07.619 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:08:09.410 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:11:38.151 35478== 2024-03-25T18:08:09.410 INFO:tasks.ceph.osd.2.smithi018.stderr:==00:00:11:38.151 35478== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:08:09.451 INFO:tasks.ceph.osd.2.smithi018.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:08:13.720 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:08:13.720 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi018 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:08:13.720 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:08:13.721 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:08:13.721 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:08:14.478 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:12:12.271 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:14.484 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:12:12.276 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:14.512 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:12:12.305 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:14.541 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:12:12.290 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:14.544 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:12:12.337 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:14.547 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:12:12.296 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:14.550 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:12:12.342 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:14.571 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:12:12.320 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:14.571 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:12:12.364 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:15.639 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:11:44.375 35479== 2024-03-25T18:08:15.639 INFO:tasks.ceph.osd.3.smithi018.stderr:==00:00:11:44.375 35479== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:08:15.679 INFO:tasks.ceph.osd.3.smithi018.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:08:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:08:19.822 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi018 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:08:19.823 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:08:19.823 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:08:19.823 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:08:21.902 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:11:50.646 35735== 2024-03-25T18:08:21.902 INFO:tasks.ceph.osd.4.smithi169.stderr:==00:00:11:50.646 35735== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:08:21.943 INFO:tasks.ceph.osd.4.smithi169.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:08:25.924 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:08:25.925 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:08:25.925 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:08:25.926 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:08:25.926 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:08:27.901 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:11:56.649 35734== 2024-03-25T18:08:27.901 INFO:tasks.ceph.osd.5.smithi169.stderr:==00:00:11:56.649 35734== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:08:27.935 INFO:tasks.ceph.osd.5.smithi169.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:08:32.028 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:08:32.028 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:08:32.029 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:08:32.029 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:08:32.029 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:08:33.680 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:12:02.438 35733== 2024-03-25T18:08:33.680 INFO:tasks.ceph.osd.6.smithi169.stderr:==00:00:12:02.438 35733== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:08:33.716 INFO:tasks.ceph.osd.6.smithi169.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:08:34.546 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:12:32.338 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:34.550 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:12:32.343 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:34.565 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:12:32.358 35121== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:34.593 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:12:32.386 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:34.595 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:12:32.344 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:34.597 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:12:32.390 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:34.599 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:12:32.347 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:34.615 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:12:32.364 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:34.616 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:12:32.408 35124== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:08:38.131 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:08:38.131 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:08:38.131 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:08:38.132 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:08:39.695 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:12:08.434 35736== 2024-03-25T18:08:39.695 INFO:tasks.ceph.osd.7.smithi169.stderr:==00:00:12:08.434 35736== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:08:39.725 INFO:tasks.ceph.osd.7.smithi169.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:08:44.233 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:08:44.233 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:08:44.233 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:08:44.233 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:08:44.234 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:08:44.234 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:08:44.260 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:08:44.261 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:08:44.261 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:08:44.287 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:08:44.287 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:08:44.288 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:08:44.288 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:08:44.317 INFO:tasks.ceph.mon.a.smithi018.stderr:2024-03-25T18:08:44.304+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35090) UID: 0 2024-03-25T18:08:44.317 INFO:tasks.ceph.mon.a.smithi018.stderr:2024-03-25T18:08:44.306+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:08:44.993 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:12:42.785 35121== 2024-03-25T18:08:44.993 INFO:tasks.ceph.mon.a.smithi018.stderr:==00:00:12:42.785 35121== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:08:45.003 INFO:tasks.ceph.mon.a.smithi018.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:08:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:08:50.390 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi018 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:08:50.391 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:08:50.391 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:08:50.391 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:08:50.407 INFO:tasks.ceph.mon.c.smithi018.stderr:2024-03-25T18:08:50.394+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35092) UID: 0 2024-03-25T18:08:50.408 INFO:tasks.ceph.mon.c.smithi018.stderr:2024-03-25T18:08:50.396+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:08:50.831 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:12:48.623 35124== 2024-03-25T18:08:50.831 INFO:tasks.ceph.mon.c.smithi018.stderr:==00:00:12:48.623 35124== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:08:50.840 INFO:tasks.ceph.mon.c.smithi018.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:08:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:08:56.493 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi018 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:08:56.493 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:08:56.494 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:08:56.494 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:08:56.511 INFO:tasks.ceph.mon.b.smithi169.stderr:2024-03-25T18:08:56.497+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 35185) UID: 0 2024-03-25T18:08:56.512 INFO:tasks.ceph.mon.b.smithi169.stderr:2024-03-25T18:08:56.499+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T18:09:24.811 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:13:22.559 35189== 2024-03-25T18:09:24.811 INFO:tasks.ceph.mon.b.smithi169.stderr:==00:00:13:22.559 35189== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:09:24.820 INFO:tasks.ceph.mon.b.smithi169.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:09:27.000 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:09:27.000 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:09:27.000 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:09:27.001 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:09:27.001 DEBUG:teuthology.orchestra.run.smithi018:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:09:27.052 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T18:09:27.052 DEBUG:teuthology.orchestra.run.smithi018:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:09:27.105 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T18:09:27.105 DEBUG:teuthology.orchestra.run.smithi018:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:09:27.163 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T18:09:27.163 DEBUG:teuthology.orchestra.run.smithi018:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:09:27.213 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T18:09:27.213 DEBUG:teuthology.orchestra.run.smithi018:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:09:27.293 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T18:09:27.293 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:09:27.471 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T18:09:27.471 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:09:27.549 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T18:09:27.549 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:09:27.616 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T18:09:27.616 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:09:27.693 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:09:27.695 DEBUG:teuthology.misc:Transferring archived files from smithi018:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621241/data/mon.a.tgz 2024-03-25T18:09:27.695 DEBUG:teuthology.orchestra.run.smithi018:> mktemp 2024-03-25T18:09:27.711 INFO:teuthology.orchestra.run.smithi018.stdout:/tmp/tmp.t9vRuhwu3w 2024-03-25T18:09:27.712 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.t9vRuhwu3w 2024-03-25T18:09:27.815 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0666 /tmp/tmp.t9vRuhwu3w 2024-03-25T18:09:27.900 DEBUG:teuthology.orchestra.remote:smithi018:/tmp/tmp.t9vRuhwu3w is 134KB 2024-03-25T18:09:27.952 DEBUG:teuthology.orchestra.run.smithi018:> rm -fr /tmp/tmp.t9vRuhwu3w 2024-03-25T18:09:27.967 DEBUG:teuthology.misc:Transferring archived files from smithi018:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621241/data/mon.c.tgz 2024-03-25T18:09:27.967 DEBUG:teuthology.orchestra.run.smithi018:> mktemp 2024-03-25T18:09:28.020 INFO:teuthology.orchestra.run.smithi018.stdout:/tmp/tmp.Mz8xPZy4It 2024-03-25T18:09:28.021 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Mz8xPZy4It 2024-03-25T18:09:28.110 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0666 /tmp/tmp.Mz8xPZy4It 2024-03-25T18:09:28.194 DEBUG:teuthology.orchestra.remote:smithi018:/tmp/tmp.Mz8xPZy4It is 134KB 2024-03-25T18:09:28.245 DEBUG:teuthology.orchestra.run.smithi018:> rm -fr /tmp/tmp.Mz8xPZy4It 2024-03-25T18:09:28.258 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621241/data/mon.b.tgz 2024-03-25T18:09:28.259 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-03-25T18:09:28.276 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.5LUKxYKoCl 2024-03-25T18:09:28.276 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.5LUKxYKoCl 2024-03-25T18:09:28.387 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.5LUKxYKoCl 2024-03-25T18:09:28.474 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.5LUKxYKoCl is 134KB 2024-03-25T18:09:28.497 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.5LUKxYKoCl 2024-03-25T18:09:28.511 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:09:28.511 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:09:28.514 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:09:28.812 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:09:29.055 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:09:29.057 DEBUG:teuthology.misc:Transferring archived files from smithi018:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621241/remote/smithi018/crash 2024-03-25T18:09:29.058 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:09:29.085 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621241/remote/smithi169/crash 2024-03-25T18:09:29.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:09:29.112 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:09:29.113 DEBUG:teuthology.orchestra.run.smithi018:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:09:29.130 DEBUG:teuthology.orchestra.run.smithi169:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:09:29.154 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41452.log 2024-03-25T18:09:29.154 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40232.log 2024-03-25T18:09:29.201 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:09:29.202 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35543.log 2024-03-25T18:09:29.202 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:09:29.202 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.35543.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:09:29.202 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:09:29.202 INFO:teuthology.orchestra.run.smithi169.stderr: 87.6% 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz -- replaced with /var/log/ceph/ceph-client.admin.35543.log.gz 2024-03-25T18:09:29.202 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T18:09:29.202 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:09:29.202 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/valgrind/osd.5.log: /var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:09:29.202 INFO:teuthology.orchestra.run.smithi169.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:09:29.202 INFO:teuthology.orchestra.run.smithi169.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:09:29.202 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35343.log 2024-03-25T18:09:29.203 INFO:teuthology.orchestra.run.smithi169.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:09:29.203 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35463.log 2024-03-25T18:09:29.203 INFO:teuthology.orchestra.run.smithi169.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:09:29.203 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.35343.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:09:29.203 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.35463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35594.log 2024-03-25T18:09:29.203 INFO:teuthology.orchestra.run.smithi169.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35343.log.gz 2024-03-25T18:09:29.203 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:09:29.203 INFO:teuthology.orchestra.run.smithi169.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35463.log.gz 2024-03-25T18:09:29.203 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35503.log 2024-03-25T18:09:29.203 INFO:teuthology.orchestra.run.smithi169.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:09:29.203 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.35594.log: /var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:09:29.203 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.35503.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:09:29.203 INFO:teuthology.orchestra.run.smithi169.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35594.log.gz 2024-03-25T18:09:29.203 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35423.log 2024-03-25T18:09:29.203 INFO:teuthology.orchestra.run.smithi169.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35503.log.gz 2024-03-25T18:09:29.203 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35383.log 2024-03-25T18:09:29.204 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.35423.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:09:29.204 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.35383.log: gzip -5 --verbose 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35423.log.gz 2024-03-25T18:09:29.204 INFO:teuthology.orchestra.run.smithi169.stderr: -- /var/log/ceph/ceph-client.admin.35634.log 2024-03-25T18:09:29.204 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-mgr.x.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35383.log.gz 2024-03-25T18:09:29.204 INFO:teuthology.orchestra.run.smithi169.stderr: -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:09:29.204 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.35634.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:09:29.204 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35634.log.gz 2024-03-25T18:09:29.204 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-mon.b.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:09:29.204 INFO:teuthology.orchestra.run.smithi169.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:09:29.220 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39253.log 2024-03-25T18:09:29.220 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40232.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:09:29.220 INFO:teuthology.orchestra.run.smithi018.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41452.log.gz 2024-03-25T18:09:29.221 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38781.log 2024-03-25T18:09:29.221 INFO:teuthology.orchestra.run.smithi018.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40232.log.gz 2024-03-25T18:09:29.221 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-mon.a.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39253.log.gz 2024-03-25T18:09:29.221 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41803.log 2024-03-25T18:09:29.221 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.38781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41959.log 2024-03-25T18:09:29.221 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41686.log 2024-03-25T18:09:29.221 INFO:teuthology.orchestra.run.smithi018.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38781.log.gz 2024-03-25T18:09:29.222 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39919.log 2024-03-25T18:09:29.222 INFO:teuthology.orchestra.run.smithi018.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41803.log.gz 2024-03-25T18:09:29.222 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41686.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41959.log.gz 2024-03-25T18:09:29.222 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42193.log 2024-03-25T18:09:29.222 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39919.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41686.log.gz 2024-03-25T18:09:29.222 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39959.log 2024-03-25T18:09:29.222 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41062.log 2024-03-25T18:09:29.223 INFO:teuthology.orchestra.run.smithi018.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39919.log.gz 2024-03-25T18:09:29.223 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39841.log 2024-03-25T18:09:29.223 INFO:teuthology.orchestra.run.smithi018.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42193.log.gz 2024-03-25T18:09:29.223 INFO:teuthology.orchestra.run.smithi018.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39959.log.gz 2024-03-25T18:09:29.223 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35469.log 2024-03-25T18:09:29.223 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39841.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:09:29.224 INFO:teuthology.orchestra.run.smithi018.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41062.log.gz 2024-03-25T18:09:29.224 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.35469.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:09:29.224 INFO:teuthology.orchestra.run.smithi018.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39841.log.gz 2024-03-25T18:09:29.224 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/valgrind/mon.c.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.35469.log.gz 2024-03-25T18:09:29.224 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:09:29.224 INFO:teuthology.orchestra.run.smithi018.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:09:29.224 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:09:29.224 INFO:teuthology.orchestra.run.smithi018.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:09:29.224 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:09:29.224 INFO:teuthology.orchestra.run.smithi018.stderr: 72.1%/var/log/ceph/valgrind/osd.1.log: -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:09:29.225 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:09:29.225 INFO:teuthology.orchestra.run.smithi018.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:09:29.225 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39175.log 2024-03-25T18:09:29.225 INFO:teuthology.orchestra.run.smithi018.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:09:29.225 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39566.log 2024-03-25T18:09:29.225 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39175.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:09:29.225 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39136.log 2024-03-25T18:09:29.225 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39645.log 2024-03-25T18:09:29.225 INFO:teuthology.orchestra.run.smithi018.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39175.log.gz 2024-03-25T18:09:29.225 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40906.log 2024-03-25T18:09:29.225 INFO:teuthology.orchestra.run.smithi018.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39566.log.gz 2024-03-25T18:09:29.225 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40945.log 2024-03-25T18:09:29.225 INFO:teuthology.orchestra.run.smithi018.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39136.log.gz 2024-03-25T18:09:29.225 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40076.log 2024-03-25T18:09:29.226 INFO:teuthology.orchestra.run.smithi018.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39645.log.gz 2024-03-25T18:09:29.226 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36470.log 2024-03-25T18:09:29.226 INFO:teuthology.orchestra.run.smithi018.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40906.log.gz 2024-03-25T18:09:29.226 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40633.log 2024-03-25T18:09:29.226 INFO:teuthology.orchestra.run.smithi018.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40945.log.gz 2024-03-25T18:09:29.226 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.36470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39605.log 2024-03-25T18:09:29.226 INFO:teuthology.orchestra.run.smithi018.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40076.log.gz 2024-03-25T18:09:29.226 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40633.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36470.log.gz 2024-03-25T18:09:29.226 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40282.log 2024-03-25T18:09:29.226 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39762.log 2024-03-25T18:09:29.226 INFO:teuthology.orchestra.run.smithi018.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40633.log.gz 2024-03-25T18:09:29.226 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40867.log 2024-03-25T18:09:29.226 INFO:teuthology.orchestra.run.smithi018.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39605.log.gz 2024-03-25T18:09:29.227 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39762.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40282.log.gz 2024-03-25T18:09:29.227 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40037.log 2024-03-25T18:09:29.227 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39370.log 2024-03-25T18:09:29.227 INFO:teuthology.orchestra.run.smithi018.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39762.log.gz 2024-03-25T18:09:29.227 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40037.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40867.log.gz 2024-03-25T18:09:29.227 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42037.log 2024-03-25T18:09:29.227 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40193.log 2024-03-25T18:09:29.227 INFO:teuthology.orchestra.run.smithi018.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40037.log.gz 2024-03-25T18:09:29.227 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35550.log 2024-03-25T18:09:29.227 INFO:teuthology.orchestra.run.smithi018.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39370.log.gz 2024-03-25T18:09:29.227 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40193.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42037.log.gz 2024-03-25T18:09:29.227 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41764.log 2024-03-25T18:09:29.227 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.35550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41569.log 2024-03-25T18:09:29.227 INFO:teuthology.orchestra.run.smithi018.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40193.log.gz 2024-03-25T18:09:29.227 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41647.log 2024-03-25T18:09:29.228 INFO:teuthology.orchestra.run.smithi018.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.35550.log.gz 2024-03-25T18:09:29.228 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41569.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41764.log.gz 2024-03-25T18:09:29.228 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:09:29.228 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41647.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41569.log.gz 2024-03-25T18:09:29.228 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41296.log 2024-03-25T18:09:29.228 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40789.log 2024-03-25T18:09:29.228 INFO:teuthology.orchestra.run.smithi018.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41647.log.gz 2024-03-25T18:09:29.228 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41491.log 2024-03-25T18:09:29.228 INFO:teuthology.orchestra.run.smithi018.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:09:29.228 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42350.log 2024-03-25T18:09:29.228 INFO:teuthology.orchestra.run.smithi018.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41296.log.gz 2024-03-25T18:09:29.228 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40360.log 2024-03-25T18:09:29.229 INFO:teuthology.orchestra.run.smithi018.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40789.log.gz 2024-03-25T18:09:29.229 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41491.log: /var/log/ceph/ceph-client.admin.42350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40115.log 2024-03-25T18:09:29.229 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40360.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29281.log 2024-03-25T18:09:29.229 INFO:teuthology.orchestra.run.smithi018.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41491.log.gz 2024-03-25T18:09:29.229 INFO:teuthology.orchestra.run.smithi018.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42350.log.gz 2024-03-25T18:09:29.229 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41920.log 2024-03-25T18:09:29.229 INFO:teuthology.orchestra.run.smithi018.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40360.log.gz 2024-03-25T18:09:29.229 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph.tmp-client.admin.29281.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29281.log.gz 2024-03-25T18:09:29.229 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- 86.8% /var/log/ceph/ceph-client.admin.41140.log 2024-03-25T18:09:29.230 INFO:teuthology.orchestra.run.smithi018.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40115.log.gz 2024-03-25T18:09:29.230 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40516.log 2024-03-25T18:09:29.230 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40984.log 2024-03-25T18:09:29.230 INFO:teuthology.orchestra.run.smithi018.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41920.log.gz 2024-03-25T18:09:29.230 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40516.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41140.log.gz 2024-03-25T18:09:29.230 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42115.log 2024-03-25T18:09:29.230 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39331.log 2024-03-25T18:09:29.230 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40984.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40516.log.gz 2024-03-25T18:09:29.230 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42232.log 2024-03-25T18:09:29.230 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39331.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:09:29.230 INFO:teuthology.orchestra.run.smithi018.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.42115.log.gz 2024-03-25T18:09:29.230 INFO:teuthology.orchestra.run.smithi018.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40984.log.gz 2024-03-25T18:09:29.230 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41023.log 2024-03-25T18:09:29.231 INFO:teuthology.orchestra.run.smithi018.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39331.log.gz 2024-03-25T18:09:29.231 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42311.log 2024-03-25T18:09:29.231 INFO:teuthology.orchestra.run.smithi018.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42232.log.gz 2024-03-25T18:09:29.231 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41179.log 2024-03-25T18:09:29.231 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42076.log 2024-03-25T18:09:29.231 INFO:teuthology.orchestra.run.smithi018.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41023.log.gz 2024-03-25T18:09:29.231 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42311.log: /var/log/ceph/ceph-client.admin.41179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40672.log 2024-03-25T18:09:29.231 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42076.log: 86.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42272.log 2024-03-25T18:09:29.231 INFO:teuthology.orchestra.run.smithi018.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42311.log.gz 2024-03-25T18:09:29.231 INFO:teuthology.orchestra.run.smithi018.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41179.log.gz 2024-03-25T18:09:29.231 INFO:teuthology.orchestra.run.smithi018.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42076.log.gz 2024-03-25T18:09:29.231 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41218.log 2024-03-25T18:09:29.231 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40399.log 2024-03-25T18:09:29.231 INFO:teuthology.orchestra.run.smithi018.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40672.log.gz 2024-03-25T18:09:29.231 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40154.log 2024-03-25T18:09:29.232 INFO:teuthology.orchestra.run.smithi018.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42272.log.gz 2024-03-25T18:09:29.232 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40828.log 2024-03-25T18:09:29.232 INFO:teuthology.orchestra.run.smithi018.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41218.log.gz 2024-03-25T18:09:29.232 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42154.log 2024-03-25T18:09:29.232 INFO:teuthology.orchestra.run.smithi018.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40399.log.gz 2024-03-25T18:09:29.232 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40438.log 2024-03-25T18:09:29.232 INFO:teuthology.orchestra.run.smithi018.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40154.log.gz 2024-03-25T18:09:29.232 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.42154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39448.log 2024-03-25T18:09:29.232 INFO:teuthology.orchestra.run.smithi018.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40828.log.gz 2024-03-25T18:09:29.232 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41413.log 2024-03-25T18:09:29.232 INFO:teuthology.orchestra.run.smithi018.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42154.log.gz 2024-03-25T18:09:29.232 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39448.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:09:29.232 INFO:teuthology.orchestra.run.smithi018.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40438.log.gz 2024-03-25T18:09:29.232 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39488.log 2024-03-25T18:09:29.232 INFO:teuthology.orchestra.run.smithi018.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39448.log.gz 2024-03-25T18:09:29.233 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40477.log 2024-03-25T18:09:29.233 INFO:teuthology.orchestra.run.smithi018.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41413.log.gz 2024-03-25T18:09:29.233 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log 2024-03-25T18:09:29.233 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40477.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39488.log.gz 2024-03-25T18:09:29.233 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41998.log 2024-03-25T18:09:29.233 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40711.log 2024-03-25T18:09:29.233 INFO:teuthology.orchestra.run.smithi018.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40477.log.gz 2024-03-25T18:09:29.233 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39292.log 2024-03-25T18:09:29.233 INFO:teuthology.orchestra.run.smithi018.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40594.log.gz 2024-03-25T18:09:29.233 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40711.log: gzip 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41998.log.gz 2024-03-25T18:09:29.233 INFO:teuthology.orchestra.run.smithi018.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39095.log 2024-03-25T18:09:29.233 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39292.log: gzip 87.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.40711.log.gz -- 2024-03-25T18:09:29.233 INFO:teuthology.orchestra.run.smithi018.stderr: /var/log/ceph/ceph-client.admin.39880.log 2024-03-25T18:09:29.233 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35132.log 2024-03-25T18:09:29.233 INFO:teuthology.orchestra.run.smithi018.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39292.log.gz 2024-03-25T18:09:29.233 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41608.log 2024-03-25T18:09:29.234 INFO:teuthology.orchestra.run.smithi018.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39095.log.gz 2024-03-25T18:09:29.234 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.35132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40321.log 2024-03-25T18:09:29.234 INFO:teuthology.orchestra.run.smithi018.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39880.log.gz 2024-03-25T18:09:29.234 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39055.log 2024-03-25T18:09:29.234 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40321.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.35132.log.gz 2024-03-25T18:09:29.234 INFO:teuthology.orchestra.run.smithi018.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41608.log.gz 2024-03-25T18:09:29.234 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:09:29.234 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41842.log 2024-03-25T18:09:29.234 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39055.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40321.log.gz 2024-03-25T18:09:29.234 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37411.log 2024-03-25T18:09:29.234 INFO:teuthology.orchestra.run.smithi018.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39055.log.gz 2024-03-25T18:09:29.234 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41101.log 2024-03-25T18:09:29.234 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.37411.log: gzip 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41842.log.gz 2024-03-25T18:09:29.235 INFO:teuthology.orchestra.run.smithi018.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39802.log 2024-03-25T18:09:29.235 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41101.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37411.log.gz 2024-03-25T18:09:29.235 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35669.log 2024-03-25T18:09:29.235 INFO:teuthology.orchestra.run.smithi018.stderr: 86.3%/var/log/ceph/ceph-client.admin.39802.log: -- replaced with /var/log/ceph/ceph-client.admin.41101.log.gz 2024-03-25T18:09:29.235 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41257.log 2024-03-25T18:09:29.235 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.35669.log: gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39802.log.gz 2024-03-25T18:09:29.235 INFO:teuthology.orchestra.run.smithi018.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40555.log 2024-03-25T18:09:29.235 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41257.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35669.log.gz 2024-03-25T18:09:29.235 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41335.log 2024-03-25T18:09:29.235 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40555.log: 86.5%gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:09:29.235 INFO:teuthology.orchestra.run.smithi018.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41257.log.gz 2024-03-25T18:09:29.235 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41530.log 2024-03-25T18:09:29.236 INFO:teuthology.orchestra.run.smithi018.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40555.log.gz 2024-03-25T18:09:29.236 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41881.log 2024-03-25T18:09:29.236 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-osd.0.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41335.log.gz 2024-03-25T18:09:29.236 INFO:teuthology.orchestra.run.smithi018.stderr:gzip/var/log/ceph/ceph-client.admin.41530.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40750.log 2024-03-25T18:09:29.236 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39527.log 2024-03-25T18:09:29.236 INFO:teuthology.orchestra.run.smithi018.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41530.log.gz 2024-03-25T18:09:29.236 INFO:teuthology.orchestra.run.smithi018.stderr:gzip 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41881.log.gz 2024-03-25T18:09:29.236 INFO:teuthology.orchestra.run.smithi018.stderr: -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:09:29.236 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40750.log: /var/log/ceph/ceph-client.admin.39527.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:09:29.236 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39527.log.gz 2024-03-25T18:09:29.236 INFO:teuthology.orchestra.run.smithi018.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40750.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39214.log 2024-03-25T18:09:29.236 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T18:09:29.236 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41725.log 2024-03-25T18:09:29.236 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.39214.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39214.log.gz 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:09:29.236 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T18:09:29.237 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39723.log 2024-03-25T18:09:29.237 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39684.log 2024-03-25T18:09:29.237 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39723.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41725.log.gz 2024-03-25T18:09:29.237 INFO:teuthology.orchestra.run.smithi018.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39723.log.gz 2024-03-25T18:09:29.237 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39998.log 2024-03-25T18:09:29.237 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39409.log 2024-03-25T18:09:29.237 INFO:teuthology.orchestra.run.smithi018.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39684.log.gz 2024-03-25T18:09:29.237 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41374.log 2024-03-25T18:09:29.237 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39409.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39998.log.gz 2024-03-25T18:09:29.237 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:09:29.237 INFO:teuthology.orchestra.run.smithi018.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39409.log.gz 2024-03-25T18:09:29.237 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41374.log: /var/log/ceph/ceph-mon.c.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41374.log.gz 2024-03-25T18:09:29.260 INFO:teuthology.orchestra.run.smithi018.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:09:29.991 INFO:teuthology.orchestra.run.smithi018.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:09:30.051 INFO:teuthology.orchestra.run.smithi169.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:09:30.057 INFO:teuthology.orchestra.run.smithi169.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:09:30.102 INFO:teuthology.orchestra.run.smithi169.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:09:30.288 INFO:teuthology.orchestra.run.smithi018.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:09:30.291 INFO:teuthology.orchestra.run.smithi018.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:09:30.320 INFO:teuthology.orchestra.run.smithi169.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:09:30.374 INFO:teuthology.orchestra.run.smithi018.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:09:34.023 INFO:teuthology.orchestra.run.smithi018.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:09:34.365 INFO:teuthology.orchestra.run.smithi169.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:09:34.367 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-03-25T18:09:34.367 INFO:teuthology.orchestra.run.smithi169.stderr:real 0m5.199s 2024-03-25T18:09:34.367 INFO:teuthology.orchestra.run.smithi169.stderr:user 0m8.713s 2024-03-25T18:09:34.367 INFO:teuthology.orchestra.run.smithi169.stderr:sys 0m0.300s 2024-03-25T18:09:34.646 INFO:teuthology.orchestra.run.smithi018.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:09:34.649 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-25T18:09:34.649 INFO:teuthology.orchestra.run.smithi018.stderr:real 0m5.507s 2024-03-25T18:09:34.649 INFO:teuthology.orchestra.run.smithi018.stderr:user 0m14.232s 2024-03-25T18:09:34.649 INFO:teuthology.orchestra.run.smithi018.stderr:sys 0m0.515s 2024-03-25T18:09:34.649 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:09:34.649 DEBUG:teuthology.misc:Transferring archived files from smithi018:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621241/remote/smithi018/log 2024-03-25T18:09:34.650 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:09:36.900 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621241/remote/smithi169/log 2024-03-25T18:09:36.901 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:09:38.325 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:09:38.525 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=53a45e5be78b48edb9b0b40ab4aa4e17 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:09:38.529 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:09:38.539 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:09:38.589 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:09:38.590 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:09:38.603 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:09:38.686 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:09:38.695 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:09:38.696 DEBUG:teuthology.orchestra.run.smithi018:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:09:38.698 DEBUG:teuthology.orchestra.run.smithi169:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:09:38.712 INFO:teuthology.orchestra.run.smithi018.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:09:38.715 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:09:38.718 INFO:teuthology.orchestra.run.smithi018.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:09:38.718 INFO:teuthology.orchestra.run.smithi018.stdout:=============================================================================== 2024-03-25T18:09:38.718 INFO:teuthology.orchestra.run.smithi018.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +565us[ +565us] +/- 85ms 2024-03-25T18:09:38.718 INFO:teuthology.orchestra.run.smithi018.stdout:^* hv02.front.sepia.ceph.com 3 6 377 52 -1242us[-1265us] +/- 66ms 2024-03-25T18:09:38.718 INFO:teuthology.orchestra.run.smithi018.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +856us[ +856us] +/- 72ms 2024-03-25T18:09:38.718 INFO:teuthology.orchestra.run.smithi018.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:09:38.720 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:09:38.720 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-03-25T18:09:38.720 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 49 +553us[ +564us] +/- 85ms 2024-03-25T18:09:38.720 INFO:teuthology.orchestra.run.smithi169.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -1249us[-1238us] +/- 67ms 2024-03-25T18:09:38.721 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 +862us[ +873us] +/- 71ms 2024-03-25T18:09:38.721 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:09:38.721 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:09:38.732 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:09:38.733 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:09:38.746 DEBUG:teuthology.orchestra.run.smithi018:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:09:38.831 DEBUG:teuthology.orchestra.run.smithi169:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:09:38.902 DEBUG:teuthology.orchestra.run.smithi018:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:09:38.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:09:38.934 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:09:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:09:38.966 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:09:38.974 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:09:38.985 INFO:teuthology.task.internal:Duration was 7369.998650 seconds 2024-03-25T18:09:38.986 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:09:38.997 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:09:38.997 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:09:38.999 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:09:39.042 INFO:teuthology.orchestra.run.smithi018.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:09:39.053 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:09:39.319 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:09:39.319 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi018.front.sepia.ceph.com 2024-03-25T18:09:39.320 DEBUG:teuthology.orchestra.run.smithi018:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:09:39.340 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi169.front.sepia.ceph.com 2024-03-25T18:09:39.341 DEBUG:teuthology.orchestra.run.smithi169:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:09:39.367 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:09:39.367 DEBUG:teuthology.orchestra.run.smithi018:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:09:39.384 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:09:39.438 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:09:39.438 DEBUG:teuthology.orchestra.run.smithi018:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:09:39.454 DEBUG:teuthology.orchestra.run.smithi169:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:09:39.581 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:09:39.591 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:09:39.591 DEBUG:teuthology.orchestra.run.smithi018:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:09:39.618 DEBUG:teuthology.orchestra.run.smithi169:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:09:39.645 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:09:39.655 DEBUG:teuthology.orchestra.run.smithi018:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:09:39.660 DEBUG:teuthology.orchestra.run.smithi169:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:09:39.683 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern = core 2024-03-25T18:09:39.711 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = core 2024-03-25T18:09:39.744 DEBUG:teuthology.orchestra.run.smithi018:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:09:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:09:39.759 DEBUG:teuthology.orchestra.run.smithi169:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:09:39.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:09:39.799 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:09:39.810 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:09:39.811 DEBUG:teuthology.misc:Transferring archived files from smithi018:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621241/remote/smithi018 2024-03-25T18:09:39.811 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:09:39.846 DEBUG:teuthology.misc:Transferring archived files from smithi169:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621241/remote/smithi169 2024-03-25T18:09:39.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:09:39.882 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:09:39.882 DEBUG:teuthology.orchestra.run.smithi018:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:09:39.885 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:09:39.931 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:09:39.941 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:09:39.941 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:09:39.951 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:09:39.952 DEBUG:teuthology.orchestra.run.smithi018:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:09:39.954 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:09:39.965 INFO:teuthology.orchestra.run.smithi018.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:09 /home/ubuntu/cephtest 2024-03-25T18:09:39.986 INFO:teuthology.orchestra.run.smithi169.stdout: 265594 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:09 /home/ubuntu/cephtest 2024-03-25T18:09:39.987 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:09:39.997 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:09:40.017 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:09:40.042 DEBUG:teuthology.nuke:shortname: smithi018 2024-03-25T18:09:40.042 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:09:40.066 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi018.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621241', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 15:57:59.039127', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM5iihOi7mgKna/7JMuqrPgo58gKUIVlTPVjbWJCGi0Pig7Q+zNrm0Y3aQEAt+gWMd16I3LFzvsidKHn5+fpqlU='} 2024-03-25T18:09:40.069 DEBUG:teuthology.nuke:shortname: smithi169 2024-03-25T18:09:40.070 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:09:40.092 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621241', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 15:57:59.041045', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGqazL4b6MWoRUTI63i1G8u0ySRHWPcHkfVSpRawEMRw0lsdpluWX2Sf71VruWn3bWzc89YLVmC5mUHvNm88rcQ='} 2024-03-25T18:09:40.120 INFO:teuthology.orchestra.console.smithi018:Power off 2024-03-25T18:09:40.120 DEBUG:teuthology.orchestra.console.smithi018:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:09:40.145 DEBUG:teuthology.orchestra.console.smithi018:power off output: Chassis Power Control: Down/Off 2024-03-25T18:09:40.146 DEBUG:teuthology.orchestra.console.smithi018:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:09:40.160 INFO:teuthology.orchestra.console.smithi169:Power off 2024-03-25T18:09:40.161 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:09:40.169 DEBUG:teuthology.orchestra.console.smithi018:check power output: Chassis Power is on 2024-03-25T18:09:40.183 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2024-03-25T18:09:40.183 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:09:40.207 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-25T18:09:44.170 DEBUG:teuthology.orchestra.console.smithi018:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:09:44.208 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:09:44.281 DEBUG:teuthology.orchestra.console.smithi018:check power output: Chassis Power is on 2024-03-25T18:09:44.318 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-25T18:09:48.287 DEBUG:teuthology.orchestra.console.smithi018:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:09:48.319 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:09:48.397 DEBUG:teuthology.orchestra.console.smithi018:check power output: Chassis Power is off 2024-03-25T18:09:48.428 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-03-25T18:09:48.499 INFO:teuthology.orchestra.console.smithi018:Power off completed 2024-03-25T18:09:52.433 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:09:52.542 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2024-03-25T18:09:52.643 INFO:teuthology.orchestra.console.smithi169:Power off completed 2024-03-25T18:09:52.747 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7369.998649597168 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=53a45e5be78b48edb9b0b40ab4aa4e17 status: fail success: false 2024-03-25T18:09:52.748 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:09:52.847 INFO:teuthology.run:FAIL